Jobs that create jobs

Over the last few years, there has been a push for more “* as code” with Jenkins configuration. You can now specify job config using a Jenkinsfile, allowing auditing and code reviews, as well as a backup.

Combined with the Job DSL plugin, this makes it possible to create a seed job (using another Jenkinsfile, naturally) that creates all the jobs for a specific project.

pipeline {
    agent any

    options {
        timestamps ()
    }

    stages {
        stage('Clean') {
            steps {
                deleteDir()
            }
        }

        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Job DSL') {
            steps {
                jobDsl targets: ['jobs/*.groovy', 'views/*.groovy'].join('\n')
            }
        }
    }
}

This will run all the groovy scripts in the jobs & views folders in this repo (once you’ve approved them).

For example:

pipelineJob("foo-main") {
    definition {
        cpsScm{
            scm {
                git {
                    remote {
                        github("examplecorp/foo", "ssh")
                    }
                    branch("main")
                }
            }
            scriptPath("Jenkinsfile")
        }
    }
    properties {
        githubProjectUrl('https://github.com/examplecorp/foo')
        pipelineTriggers {
            triggers {
                cron {
                     spec('@daily')
                }
                githubPush()
            }
        }
    }
}

And a view, to put it in:

listView('foo') {
    description('')

    jobs {
        regex('foo-.*')
    }

    columns {
        status()
        weather()
        name()
        lastSuccess()
        lastFailure()
        lastDuration()
        buildButton()
    }
}

Jenkins and oauth2_proxy

We hide Jenkins behind bitly’s oauth2_proxy, to control access using our Google accounts. After recently upgrading to Debian Jessie (amongst other things), we found that enabling security on Jenkins (using the Reverse Proxy Auth plugin) resulted in an error:

java.lang.NullPointerException
	at org.jenkinsci.plugins.reverse_proxy_auth.ReverseProxySecurityRealm$1.doFilter(ReverseProxySecurityRealm.java:435)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
...

Following the stack trace, we find ourselves here. It’s pretty obvious that the NRE is caused by u being null, but the real question is why we are in that if block at all.

It turns out that at some point the oauth proxy started sending a basic auth header, as well as the X-Forwarded ones we need. This makes the Jenkins plugin sad, when it tries to look up the user.

Unfortunately, there is currently no way to have one without the other, which is an issue for other upstream applications. Hopefully at some point that flag will be added, but until then I’ve simply deleted the offending line.

Jenkins + SSH keys

Jenkins makes it very easy to manage SSH keys. You can use the Credentials plugin to store the key, and then the SSH Agent plugin to seamlessly expose it to your build.

The downside is that now everyone with access to Jenkins has access to that key. It’s possible to use roles to restrict access through the web UI, but in our case it’s useful to allow access to the machine Jenkins is running on (for debugging purposes). And Jenkins itself has r+w privileges, so it’s all but impossible to prevent reading that file.

When the key is used for deploying to production, that’s a problem. Access to the key itself is actually useless, as it’s passphrase protected, but using the solution described above means the passphrase is stored in a credentials.xml file in $JENKINS_HOME. The file is encrypted, but reversing that is trivial.

It would be handy if the SSH Agent plugin allowed prompting for the passphrase before running a build, but that doesn’t appear to be a thing. It is possible however, to use the Parameterized Build plugin to emulate that.

This means you need to start ssh-agent yourself, and due to the fact that ssh-add doesn’t play nice with stdin, there’s some hoop jumping involved. The easiest method seems to involve using expect:

#!/bin/bash

expect << EOF
  spawn ssh-add $1
  expect "Enter passphrase"
  send "$SSH_PASSPHRASE\r"
  expect eof
EOF

Then, assuming you added a build parameter named SSH_PASSPHRASE, you can use this script after launching ssh-agent and before you need the ssh key:

eval `ssh-agent`
./ssh-add-pass ./key_file
./run_playbook