Running locust on Fargate

Locust is a programmer-friendly load testing tool (certainly compared with jmeter!). Traditionally, once you needed to generate more load than a single host could easily support, you would set up a swarm. However, if you’re willing to live without the web UI, there is another option.

Once you have a containerised version of your scripts, you can go “serverless”, and run them as a task on AWS Fargate. You can use the wizard to set up a cluster &c, or define them with cloudformation:

AWSTemplateFormatVersion: 2010-09-09


    Type: 'AWS::ECS::Cluster'
      ClusterName: ${cluster_name}

    Type: 'AWS::IAM::Role'
          - Effect: Allow
              - 'sts:AssumeRole'
      Path: /
        - PolicyName: logs
              - Effect: Allow
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: '*'
        - PolicyName: ecr
              - Effect: Allow
                  - 'ecr:BatchCheckLayerAvailability'
                  - 'ecr:GetDownloadUrlForLayer'
                  - 'ecr:BatchGetImage'
                Resource: !Join
                  - ''
                  - - 'arn:aws:ecr:'
                    - !Ref 'AWS::Region'
                    - ':'
                    - !Ref 'AWS::AccountId'
                    - ':repository/your-repo'
              - Effect: Allow
                  - 'ecr:GetAuthorizationToken'
                Resource: '*'

    Type: AWS::Logs::LogGroup
      LogGroupName: !Sub /fargate/${AWS::StackName}
      RetentionInDays: 7

    Type: 'AWS::ECS::TaskDefinition'
      Family: ${name}
      Cpu: 256
      Memory: 512
      NetworkMode: awsvpc
        - FARGATE
      ExecutionRoleArn: !Ref TaskExecutionRole
        - Name: ${name}
          Cpu: 256
          Memory: 512
          Image: ${account_id}.dkr.ecr.${region}${image_name}:latest
            LogDriver: awslogs
              awslogs-region: ${region}
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: !Ref AWS::StackName

Once the stack is created, you can run a task on the cluster:

aws ecs run-task --launch-type FARGATE --cluster ${cluster-name} --task-definition ${task-name}:${latest-revision} --network-configuration "awsvpcConfiguration={subnets=[${public-subnet-id}],securityGroups=[${security-group}],assignPublicIp='ENABLED'}" --count 1 --overrides '{"containerOverrides":[{"name":${name},"environment":[{"name":"TARGET_URL","value":${target-url}},{"name":"LOCUST_OPTS","value":"--clients=100 --no-web --only-summary --run-time=1h"}]}]}'

(You can use a public subnet/sg from the default vpc). That will spawn 100 VUs, for an hour, against your chosen target. And you can just keep adding more. Any logs from locust will be available in the AWS console.

Terminating TLS at an ALB

While there are definite benefits to having a zero trust network, it’s also convenient to outsource all the certificate management.

First you need to create a cert with ACM, either by importing it, or letting them do it (managed renewal ftw!):

    Type: AWS::CertificateManager::Certificate
      DomainName: !Ref 'DomainName'
      ValidationMethod: DNS

That in hand, you can create the LB, Listener & Target Group:

    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
      Name: foo
        - !Ref PublicSubnet1
        - ...
        - !Ref SecurityGroup
    Type: AWS::ElasticLoadBalancingV2::Listener
      LoadBalancerArn: !Ref LoadBalancer
      Port: 443
      Protocol: HTTPS
        - Type: forward
          TargetGroupArn: !Ref DefaultTargetGroup
      SslPolicy: ELBSecurityPolicy-TLS-1-2-2017-01
        - CertificateArn: !Ref Certificate
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
      Name: foo
      VpcId: !Ref Vpc
      Port: 80
      Protocol: HTTP
    Type: AWS::EC2::SecurityGroup
      VpcId: !Ref Vpc
      GroupDescription: Enable HTTPS access for LB
        - IpProtocol: tcp
          FromPort: '443'
          ToPort: '443'
          CidrIp: ''

Make sure you use a public subnet, or you won’t be able to reach the LB!

Ansible, AWS, and bastion hosts, oh my!

There’s some useful info available about using a jump host with Ansible, and AWS dynamic inventory; but either the world has changed since those were written, or my scenario is slightly different.

I defined my inventory first:

plugin: aws_ec2
  - eu-west-2
 - key: tags.Type
   separator: ''
  ansible_host: private_ip_address

(that last bit is important, otherwise the ssh config won’t work). At this point you should be able to list (or graph) the instances you want to connect to:

$ ansible-inventory -i inventories/eu-west-2.aws_ec2.yml --list

Next you need some ssh config:

Host 10.0.*.*
    ProxyCommand ssh -W %h:%p admin@

I kept it pretty minimal. The IP mask needs to match whatever you used for the subnet(s) the instances are attached to (obvs). And the login may vary depending on the image you used, if you are using the defaults.

You can then use this config when running your playbook:

ANSIBLE_SSH_ARGS="-F lon_ssh_config" ansible AppServer -i inventories/eu-west-2.aws_ec2.yml -u admin -m ping

The IP address for the jump host is hard-coded in the ssh config, which isn’t ideal. We may use a DNS record, and update that instead, if it changes; but there doesn’t seem any easy way to either get that from the inventory, or update the cname automatically.

AutoScalingGroup with a LaunchTemplate

There are plenty of examples for creating an ASG using a CloudFormation template, but those I found all used a “launch configuration“.

According to the docs, using a launch template is the new hotness, so I foolishly assumed it would be simple to adapt one to the other.

Some time later, I had a working example:

    Type: 'AWS::AutoScaling::AutoScalingGroup'
        LaunchTemplateId: !Ref LaunchTemplate
        Version: !GetAtt LaunchTemplate.LatestVersionNumber
        - Fn::ImportValue:
            Fn::Sub: '${Network1StackName}-PublicSubnetId'
        - Fn::ImportValue:
            Fn::Sub: '${Network2StackName}-PublicSubnetId'
      MinSize: 1
      MaxSize: 1
    Type: 'AWS::EC2::LaunchTemplate'
        ImageId: "..."
        InstanceType: "..."
          - Fn::ImportValue:
              Fn::Sub: '${SecurityGroupsStackName}-SshIngressSecurityGroupId'

Word chains (Part 3)

Last time, we found the possible next words. Now we want to build on that, and use that function to build a chain from the first word, to the goal word. Sounds like a job for recursion (divide and conquer)!

This time, we’ll check that the chains generated all end in the expected word:

prop_all_chains_should_include_last_word() ->
    ?FORALL({FirstWord, LastWord}, valid_words(),
            Words = word_chains:word_list(length(FirstWord)),
            Chains = word_chains:all_chains(FirstWord, LastWord, Words, length(FirstWord)),
            InvalidChains = lists:filter(fun([W|_]) -> W =/= LastWord end, Chains),
            length(InvalidChains) =:= 0

We pass in the word list (all words of the chosen length), to avoid reading the file multiple times.

all_chains(FirstWord, LastWord, Words, MaxLength) ->
    lists:sort(fun(A, B) -> length(A) =< length(B) end, all_chains(FirstWord, LastWord, Words, MaxLength, [[FirstWord]])).

all_chains(FirstWord, LastWord, Words, MaxLength, Chains) ->
    lists:append(lists:map(fun(Chain) ->
        [CurrentWord | _Rest] = Chain,
        case CurrentWord =:= LastWord of
            true -> [Chain];
            false ->
                NextWords = next_words(CurrentWord, Words),
                NewChains = compact(lists:map(fun(NewWord) ->
                    case lists:member(NewWord, Chain) of
                        false ->
                            NewChain = [NewWord | Chain],
                            case length(NewChain) > MaxLength of
                                true -> [];
                                false -> NewChain
                        true -> []
                end, NextWords)),
                all_chains(FirstWord, LastWord, Words, MaxLength, NewChains)
    end, Chains)).

Our first chain is simply the first word, e.g. [“cat”]. We then iterate over the list, and find all possible next words, and create the possible chains using those words, [[“bat”, “cat”], [“cab”, “cat”], &c …] .

If any chain ends in the target word, no more work is required. Otherwise we continue to extend, and branch, the chains. If the proposed next word already exists in the current chain, then that branch is dead (to avoid looping forever).

Once all branches have been exhausted, we return the list of valid chains, sorted by length (shortest first).

Unfortunately, while it seemed like a good idea to generate all possible chains, it turns out that some of them can be very long. So I added a max length param, to cut short further exploration.

Even with that, execution can be pretty slow; so next time we’ll do some profiling, and see if caching the possible next words will help.

Word chains (Part 2)

Previously, we laid some groundwork for generating word chains. Rather than arbitrarily returning one word, we might as well get all the words that are one letter different from the first word:

prop_next_words_should_be_near() ->
    ?FORALL({FirstWord, LastWord}, valid_words(),
            NextWords = word_chains:next_words(FirstWord),
            InvalidWords = lists:filter(fun(W) -> word_chains:get_word_distance(W, FirstWord) =/= 1 end, NextWords),
            length(InvalidWords) =:= 0

We can calculate the “word distance” using map/reduce:

get_word_distance(Word1, Word2) ->
    Differences = lists:zipwith(fun(X, Y) -> case X =:= Y of true -> 0; false -> 1 end end, Word1, Word2),
    lists:foldl(fun(D, Acc) -> Acc + D end, 0, Differences).

For each letter in Word1, we compare it with the same (position) letter in Word2, and assign a 0 if it matches and a 1 if it differs. The sum of these values tells us the difference between the 2 words.

2> word_chains:get_word_distance("cat", "cat").
3> word_chains:get_word_distance("cat", "cot").
4> word_chains:get_word_distance("cat", "cog").

Using this helper function, we can easily find all the possible next words:

next_words(FirstWord) ->
    WordList = word_list(),
    SameLengthWords = lists:filter(fun(W) -> length(W) =:= length(FirstWord) end, WordList),
    WordDistances = lists:map(fun(W) -> {W, get_word_distance(W, FirstWord)} end, SameLengthWords),
    lists:map(fun({Word, _}) -> Word end, lists:filter(fun({_, Distance}) -> Distance =:= 1 end, WordDistances)).

Almost there! Next time, we will actually start generating some word chains.

Word chains (Part 1)

I’ve recently been using the word chains kata for interviewing, and I thought it might be interesting to try using Erlang, and property testing.

The first step is to get a word list. I thought most linux distros came with a dictionary file, but my laptop only had a crack lib, which wasn’t really what I was looking for.

I had used this npm package before, so I just downloaded the text file it provides. With that hand, getting a list of words is easy:

word_list() ->
    {ok, Data} = file:read_file("words.txt"),
    binary:split(Data, [<<"\n">>], [global]).

The next step is to find all words that are one letter away from the first word. So we create a property:

prop_next_word_should_be_new() ->
    ?FORALL({FirstWord, LastWord}, valid_words(),
            NextWord = word_chains:next_word(FirstWord, LastWord),
            NextWord =/= FirstWord

For each first word/last word pair, we check that the next word is different from the first word. We also need a generator, of valid words:

valid_words() ->
    ?SUCHTHAT({FirstWord, LastWord},
        ?LET(N, choose(2, 10),
                WordList = word_chains:word_list(),
                SameLengthWords = lists:filter(fun(W) -> length(W) =:= N end, WordList),
                {random_word(SameLengthWords), random_word(SameLengthWords)}
    FirstWord =/= LastWord).

random_word(Words) ->
    lists:nth(rand:uniform(length(Words)), Words).

First we pick a random number, in the range (2, 10), and then pick 2 words of that length, from the full word list, at random. This could result in the same word being used as both first & last word, so we filter that out, using the ?SUCHTHAT macro.

For now, we can make this pass by simply returning the last word:

next_word(_FirstWord, LastWord) ->
$ ./rebar3 proper
===> Verifying dependencies...
===> Compiling word_chains
===> Testing prop_word_chains:prop_next_word_should_be_new()
OK: Passed 100 test(s).
1/1 properties passed

Boom! Next time, a more useful implementation of next word.

Grouping by time period in Redshift

We use Redshift for reporting (or, more accurately, the tool we use for reporting uses Redshift).

It’s pretty simple to calculate daily rollups, using datetrunc:

select date_trunc('day', foo) date, sum(bar) bar
from [baz]
where ...
group by date

Or just by casting to a date:

select [foo:date] date, sum(bar) bar
from [baz]
where ...
group by date

But what if you want to slice into smaller time periods?

The easiest way I’ve found, is to use to_char, and format the timestamp appropriately. e.g.

select to_char(foo, 'YYYY-MM-DD HH24:MI') date, sum(bar) bar
from [baz]
where ...
group by date

Using “free” wi-fi on Linux

I recently found myself at a train station that claimed to offer “free super-fast wi-fi”; but when I connected to it, I was stuck with a little question mark in the gnome toolbar, and the chrome dinosaur.

I’m not the first person to note this, but I did manage to get it working… eventually.

I opened up a terminal, and even though the browser was sad, I could ping:

$ ping
PING ( 56(84) bytes of data.
From _gateway ( icmp_seq=1 Destination Net Prohibited

Sort of. DNS was working, anyway. So I tried curl:

$ curl -vL ""
* Rebuilt URL to:
*   Trying
* Connected to ( port 443 (#0)
... snip ...

And it looked like I was getting the actual Google homepage, not a hijacked page, which was interesting.

At that point I took the IP address from that request, and stuck it directly in the browser, hoping to skip straight there; and now the browser decided to show me the T&C page I needed. Once I’d ticked the right box, I was online.