ETIMEDOUT connecting to pgbouncer

We use pgbouncer as a connection pooler, and in one of our production enviroments (after a recent migration) we were getting some portion of connection attempts failing with ETIMEDOUT.

Our first assumption was that it was due to some limitation of our service provider’s internal network, but they assured us that they couldn’t see any failures; and when we looked at the other end, we couldn’t either.

So it seemed to be some limitation on the client host (e.g. hitting the file descriptor limit). We had a look at some netstat data, and added some datadog tcp metrics, but nothing stood out.

At this point, there seemed to be no other option than to use tcpdump and see if we could find a reason that the connection was rejected. We fired it up:

sudo tcpdump -i eth2 -w tcpdump.log

downloaded the output, and opened it up in wireshark. Following some helpful instructions we identified some likely packets.

At this point it was starting to look like we were suffering from ephemeral port exhaustion, so we decided to experiment with running pgbouncer on the app server instead, as that would reduce the number of open sockets between the hosts.

A resounding success! I’m sure it would also be possible to tune some linux tcp options, to the same effect, but this was acceptable for us (there’s only one app server in that env).

I’m not entirely sure why we were getting a time out, rather than EADDRNOTAVAIL, but that may be due to the client library we are using to connect.

Bootstrapping node in docker

It always seems to take me a few attempts to get this right, so I’m making a note here:

docker run -it -v $PWD:/app -w /app node:12-alpine npm init

You may need to chown the package.json after (the docker daemon runs as root).

You can use the same thing to add packages:

docker run -it -v $PWD:/app -w /app node:12-alpine npm i --save foo

Disk by id

We’ve been using an (openstack based) cloud provider, that can’t guarantee a stable device name for an attached volume.

This was causing problems when used in /etc/fstab; on reboot, if the device name was incorrect, the instance would hang.

It’s pretty straight forward to use the UUID instead, with ansible:

- name: Mount vol
  become: yes
  mount:
    path: "{{ mount_point }}"
    src: "UUID={{ ansible_devices[device_name].partitions[device_name + '1'].uuid }}"
    fstype: ext4
    state: mounted

but we still needed the device_name in group vars. Our provider explained that a stable id was provided, in /dev/disk/by-id, which could be used directly for most tasks:

- name: Create a new primary partition
  parted:
    device: "/dev/disk/by-id/{{ device_id }}"
    number: 1
    state: present
  become: yes

- name: Create ext4 filesystem on vol
  become: yes
  filesystem:
    fstype: ext4
    dev: "/dev/disk/by-id/{{ device_id }}-part1"

But how do you get from the id, to the device name?

$ ls /dev/disk/by-id/
virtio-c11c38e5-7021-48d2-a  virtio-c11c38e5-7021-48d2-a-part1
"ansible_devices": {
        "vda": {
            ...
        }, 
        "vdb": {
            ...
        }, 
        "vdc": {
            ...
            "links": {
                "ids": [
                    "virtio-c11c38e5-7021-48d2-a"
                ], 
                ...
            }, 
            ...
        }
    }

This seemed like a job for json_query but, after a fruitless hour or two, I gave up and used this (slightly hacky) solution suggested on SO:

- name: Get device name
  set_fact:
    device_name: "{{ item.key }}"
  with_dict: "{{ ansible_devices }}"
  when: "(item.value.links.ids[0] | default()) == device_id"
  no_log: yes

Resetting all sequences in postgresql

It’s pretty simple to update the next value generated by a sequence, but what if you want to update them for every table?

In our case, we had been using DMS to import data, but none of the sequences were updated afterwards. So any attempt to insert a new row was doomed to failure.

To update one sequence you can call:

SELECT setval('foo.bar_id_seq', (select max(id) from foo.bar), true);

and you can get a list of tables pretty easily:

\dt *.*

but how do you put them together? My first attempt was using some vim fu (qq@q), until I realised I’d need to use a regex to capture the table name. And then I found some sequences that weren’t using the same name as the table anyway (consistency uber alles).

It’s also easy to get a list of sequences:

SELECT * FROM information_schema.sequences;

but how can you link them back to the table?

The solution is a function called pg_get_serial_sequence:

select t.schemaname, t.tablename, pg_get_serial_sequence(t.schemaname || '.' || t.tablename, c.column_name)
from pg_tables t
join information_schema.columns c on c.table_schema = t.schemaname and c.table_name = t.tablename
where t.schemaname <> 'pg_catalog' and t.schemaname <> 'information_schema' and pg_get_serial_sequence(t.schemaname || '.' || t.tablename, c.column_name) is not null;

This returns the schema, table name, and sequence name for every (non-system) table; which should be “trivial” to convert to a script updating the sequences (I considered doing that in sql, but dynamic tables aren’t easy to do).

Streaming a csv from postgresql

If you want to build an endpoint to download a csv, that could contain a large number of rows; you want to use streams, so you don’t need to hold all the data in memory before writing it.

If you are already using the pg client, it has a nifty add-on for this purpose:

const { Client } = require('pg');
const QueryStream = require('pg-query-stream');
const csvWriter = require("csv-write-stream");

module.exports = function(connectionString) {
    this.handle = function(req, res) {
        var sql = "SELECT...";
        var args = [...];

        const client = new Client({connectionString});
        client.connect().then(() => {
            var stream = new QueryStream(sql, args);
            stream.on('end', () => {
                client.end();
            });
            var query = client.query(stream);

            var writer = csvWriter();
            res.contentType("text/csv");
            writer.pipe(res);

            query.pipe(writer);
        });
    };
};

If you need to transform the data, you can add another step:

...

const transform = require('stream-transform');

            ...

            var query = client.query(stream);

            var transformer = transform(r => ({
                "User ID": r.user_id,
                "Created": r.created.toISOString(),
                ...
            }));

            ...

            query.pipe(transformer).pipe(writer);

Running locust on Fargate

Locust is a programmer-friendly load testing tool (certainly compared with jmeter!). Traditionally, once you needed to generate more load than a single host could easily support, you would set up a swarm. However, if you’re willing to live without the web UI, there is another option.

Once you have a containerised version of your scripts, you can go “serverless”, and run them as a task on AWS Fargate. You can use the wizard to set up a cluster &c, or define them with cloudformation:

AWSTemplateFormatVersion: 2010-09-09

Resources:

  Cluster:
    Type: 'AWS::ECS::Cluster'
    Properties:
      ClusterName: ${cluster_name}

  TaskExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: logs
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: '*'
        - PolicyName: ecr
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - 'ecr:BatchCheckLayerAvailability'
                  - 'ecr:GetDownloadUrlForLayer'
                  - 'ecr:BatchGetImage'
                Resource: !Join
                  - ''
                  - - 'arn:aws:ecr:'
                    - !Ref 'AWS::Region'
                    - ':'
                    - !Ref 'AWS::AccountId'
                    - ':repository/your-repo'
              - Effect: Allow
                Action:
                  - 'ecr:GetAuthorizationToken'
                Resource: '*'

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /fargate/${AWS::StackName}
      RetentionInDays: 7

  TaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Properties:
      Family: ${name}
      Cpu: 256
      Memory: 512
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !Ref TaskExecutionRole
      ContainerDefinitions:
        - Name: ${name}
          Cpu: 256
          Memory: 512
          Image: ${account_id}.dkr.ecr.${region}.amazonaws.com/${image_name}:latest
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: ${region}
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: !Ref AWS::StackName

Once the stack is created, you can run a task on the cluster:

aws ecs run-task --launch-type FARGATE --cluster ${cluster-name} --task-definition ${task-name}:${latest-revision} --network-configuration "awsvpcConfiguration={subnets=[${public-subnet-id}],securityGroups=[${security-group}],assignPublicIp='ENABLED'}" --count 1 --overrides '{"containerOverrides":[{"name":${name},"environment":[{"name":"TARGET_URL","value":${target-url}},{"name":"LOCUST_OPTS","value":"--clients=100 --no-web --only-summary --run-time=1h"}]}]}'

(You can use a public subnet/sg from the default vpc). That will spawn 100 VUs, for an hour, against your chosen target. And you can just keep adding more. Any logs from locust will be available in the AWS console.

Terminating TLS at an ALB

While there are definite benefits to having a zero trust network, it’s also convenient to outsource all the certificate management.

First you need to create a cert with ACM, either by importing it, or letting them do it (managed renewal ftw!):

  Certificate:
    Type: AWS::CertificateManager::Certificate
    Properties:
      DomainName: !Ref 'DomainName'
      ValidationMethod: DNS

That in hand, you can create the LB, Listener & Target Group:

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: foo
      Subnets:
        - !Ref PublicSubnet1
        - ...
      SecurityGroups:
        - !Ref SecurityGroup
  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Port: 443
      Protocol: HTTPS
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref DefaultTargetGroup
      SslPolicy: ELBSecurityPolicy-TLS-1-2-2017-01
      Certificates:
        - CertificateArn: !Ref Certificate
  DefaultTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: foo
      VpcId: !Ref Vpc
      Port: 80
      Protocol: HTTP
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref Vpc
      GroupDescription: Enable HTTPS access for LB
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '443'
          ToPort: '443'
          CidrIp: '0.0.0.0/0'

Make sure you use a public subnet, or you won’t be able to reach the LB!

Ansible, AWS, and bastion hosts, oh my!

There’s some useful info available about using a jump host with Ansible, and AWS dynamic inventory; but either the world has changed since those were written, or my scenario is slightly different.

I defined my inventory first:

plugin: aws_ec2
regions:
  - eu-west-2
keyed_groups:
 - key: tags.Type
   separator: ''
compose:
  ansible_host: private_ip_address

(that last bit is important, otherwise the ssh config won’t work). At this point you should be able to list (or graph) the instances you want to connect to:

$ ansible-inventory -i inventories/eu-west-2.aws_ec2.yml --list

Next you need some ssh config:

Host 10.0.*.*
    ProxyCommand ssh -W %h:%p admin@52.56.111.199

I kept it pretty minimal. The IP mask needs to match whatever you used for the subnet(s) the instances are attached to (obvs). And the login may vary depending on the image you used, if you are using the defaults.

You can then use this config when running your playbook:

ANSIBLE_SSH_ARGS="-F lon_ssh_config" ansible AppServer -i inventories/eu-west-2.aws_ec2.yml -u admin -m ping

The IP address for the jump host is hard-coded in the ssh config, which isn’t ideal. We may use a DNS record, and update that instead, if it changes; but there doesn’t seem any easy way to either get that from the inventory, or update the cname automatically.

AutoScalingGroup with a LaunchTemplate

There are plenty of examples for creating an ASG using a CloudFormation template, but those I found all used a “launch configuration“.

According to the docs, using a launch template is the new hotness, so I foolishly assumed it would be simple to adapt one to the other.

Some time later, I had a working example:

  AutoScalingGroup:
    Type: 'AWS::AutoScaling::AutoScalingGroup'
    Properties:
      LaunchTemplate:
        LaunchTemplateId: !Ref LaunchTemplate
        Version: !GetAtt LaunchTemplate.LatestVersionNumber
      VPCZoneIdentifier:
        - Fn::ImportValue:
            Fn::Sub: '${Network1StackName}-PublicSubnetId'
        - Fn::ImportValue:
            Fn::Sub: '${Network2StackName}-PublicSubnetId'
      MinSize: 1
      MaxSize: 1
  LaunchTemplate:
    Type: 'AWS::EC2::LaunchTemplate'
    Properties:
      LaunchTemplateData:
        ImageId: "..."
        InstanceType: "..."
        SecurityGroupIds:
          - Fn::ImportValue:
              Fn::Sub: '${SecurityGroupsStackName}-SshIngressSecurityGroupId'