Uploading to Strava from Arch Linux

I’ve had a MBP for work for the last year or so, so had side-stepped the misery of getting GPS data from my Forerunner 305 on to Strava. Now I’m back on Arch full time, and the plugin I was using no longer works; so it was time to find a new approach.

The first thing you will need is a copy of garmintools, which no longer seems to be available in the AUR. If you can get that built and installed, then I refer you to my previous article to get it working.

At that point, you should be able to run “garmin_save_runs”, and all the data on your device will be exported (in the current working directory!). The data is organised by date, so it should be pretty easy to find the track you want.

Unfortunately, the data is exported as a “.gmn” file, which isn’t supported by Strava; so we need to convert it. Next stop is garmin-dev. You can either clone the repo (if you know what git is), or just download the zip. It’s then as simple as pointing the tool at your file, and saving the output:

./gmn2tcx 20160801T151814.gmn > 20160801T151814.tcx

And uploading the tcx file to Strava. Happy trails!

JSON error page for Nginx

We use the ngx_http_auth_request_module to authenticate requests by calling one of our services. Unfortunately, if the request is rejected, it returns the default 401 error page:

<html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.9.10</center>
</body>
</html>

Which is a bit out of place, when the rest of our responses are JSON. A simple fix is to override it:

server {
    ...
    error_page 401 @401_json;

    location @401_json {
        default_type application/json;
        return 401 '{"error":{"message":"Unauthorised"}}';
    }
}

But remember this will be used for all 401s, not just those from the auth module.

Mapping conflicts with ELK

I recently started upgrading to a newer version of ES (4.5), and found that it refused to start:

IllegalStateException: unable to upgrade the mappings for the index

In fact, this mapping conflict was one of the things I was hoping the upgrade would solve. After a bit of reading it became clear that I would have to make some changes.

The mapping in question was a field from the logs called “level”. In the postgres logs it was a string (e.g. “INFO”), and in our application logs (using bunyan) it was an integer (40 => “WARN”).

To allow me to search using a range (e.g. level:[40 TO 60]), I was using a mutate filter to convert the string “40” to an integer, and this was the cause of the conflict.

My first thought was to copy the field before converting:

mutate {
    add_field => { "level_int" => "%{level}" }
    convert => { "level_int" => "integer" }
}

But it turns out that that’s not enough to avoid a conflict (possibly because ES guesses the type, and saw an int first?). So I went with the nuclear option, and renamed the field:

mutate {
    rename => { "level" => "level_int" }
    convert => { "level_int" => "integer" }
}

Now my new documents were conflict free. Unfortunately, the only solution provided for existing data is to export and re-import it, which I wasn’t really in the mood for.

Luckily, I’m not in any rush to upgrade, and we close indices after 30 days. So I plan to wait for a month, and hope my data is clean by then!

“Trouble parsing json”

We use Bunyan┬áin our node apps, for “structured logging”. The output json string is passed to syslog, by systemd, and then fed into ELK.

{
    "name":"foo-service",
    "hostname":"app-01",
    "pid":30988,
    "ip":"1.19.24.8",
    "requestId":"1c11f448-73f2-4efa-bc63-3de787618d49",
    "level":50,
    "err": {
        "message":"oh noes!"
    }
}

Unfortunately, if that string is longer than 2048 chars (usually a stacktrace, or html returned from a web service instead of json), then the json blob ends up split over 2 lines in syslog.

This causes ELK to barf when attempting to parse the broken lines (assuming you are parsing as json), and means you won’t see those errors in Kibana.

It is possible to detect the error parsing the error, by searching for the string “Trouble parsing json”, but that’s not really a solution.

I would prefer to see a truncated error, than have the current situation, but that means either wrapping or patching Bunyan itself.

Node.js and systemd-notify

Running a nodejs app as a systemd service is pretty simple. But the downside to this approach is that if the app dies straight away (e.g. a config file is malformed), then systemd remains unaware.

An alternative is to switch to using notify:

[Service]
Type=notify
ExecStart=/var/www/app_name/app.js
Restart=always
...

There are various ways to notify systemd that your service is ready; but the simplest is to use systemd-notify, a console wrapper around sd_notify.

this.start = function(port) {
    var deferred = Q.defer();

    server = app.listen(port, '127.0.0.1', function(error) {
        if (error) {
            deferred.reject(new Error(error));
        } else {
            logger.info('Listening on port %d', port);
            exec('systemd-notify --ready');
            deferred.resolve();
        }
        });

        return deferred.promise;
    };
};

If the process that calls systemd-notify is not the one in ExecStart (e.g. you are using cluster), you will also need to set NotifyAccess to “all”.

Now when you use systemctl to start your service, it will wait for the notification before deeming the start process complete.

Jenkins and oauth2_proxy

We hide Jenkins behind bitly’s oauth2_proxy, to control access using our Google accounts. After recently upgrading to Debian Jessie (amongst other things), we found that enabling security on Jenkins (using the Reverse Proxy Auth plugin) resulted in an error:

java.lang.NullPointerException
	at org.jenkinsci.plugins.reverse_proxy_auth.ReverseProxySecurityRealm$1.doFilter(ReverseProxySecurityRealm.java:435)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
...

Following the stack trace, we find ourselves here. It’s pretty obvious that the NRE is caused by u being null, but the real question is why we are in that if block at all.

It turns out that at some point the oauth proxy started sending a basic auth header, as well as the X-Forwarded ones we need. This makes the Jenkins plugin sad, when it tries to look up the user.

Unfortunately, there is currently no way to have one without the other, which is an issue for other upstream applications. Hopefully at some point that flag will be added, but until then I’ve simply deleted the offending line.

Error from postgresql_user

We have an ansible task that creates a postgres user (role) for vagrant:

- name: Create vagrant user
  sudo: true
  sudo_user: postgres
  postgresql_user: name=vagrant role_attr_flags=CREATEDB,CREATEUSER

which was working fine with ansible 1.9; but when we upgraded to 2.0, we started getting an error if the user already existed.

TASK [pg-vagrant : Create vagrant user] ****************************************
fatal: [default]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible-tmp-1457016468.17-9802839733620/postgresql_user\", line 2722, in \r\n    main()\r\n  File \"/tmp/ansible-tmp-1457016468.17-9802839733620/postgresql_user\", line 621, in main\r\n    changed = user_alter(cursor, module, user, password, role_attr_flags, encrypted, expires, no_password_changes)\r\n  File \"/tmp/ansible-tmp-1457016468.17-9802839733620/postgresql_user\", line 274, in user_alter\r\n    if current_role_attrs[PRIV_TO_AUTHID_COLUMN[role_attr_name]] != role_attr_value:\r\n  File \"/usr/lib/python2.7/dist-packages/psycopg2/extras.py\", line 144, in __getitem__\r\n    x = self._index[x]\r\nKeyError: 'rolcreateuser'\r\n", "msg": "MODULE FAILURE", "parsed": false}

When I checked that the user had been created successfully the first time I provisioned:

=# \dg
                                 List of roles
    Role name     |                   Attributes                   | Member of
------------------+------------------------------------------------+-----------
 postgres         | Superuser, Create role, Create DB, Replication | {}
 vagrant          | Superuser, Create DB                           | {}

I noticed that the role actually had Superuser privileges, something the documentation confirmed:

CREATEUSER
NOCREATEUSER
These clauses are an obsolete, but still accepted, spelling of SUPERUSER and NOSUPERUSER. Note that they are not equivalent to CREATEROLE as one might naively expect!

So it looks like somewhere in the toolchain (ansible, psycopg2, postgres) this is no longer supported. Substituting SUPERUSER for CREATEUSER fixed the issue.