jenkins

Jenkins seed job

Graham HayLeave a comment

In the brave new world of Jenkins as Code, you can use CasC to specify an initial job (using the Job DSL):

jobs:
  - script: >
      pipelineJob('jenkins-job-dsl') {
        definition {
          cpsScm{
            scm {
              gitSCM {
                userRemoteConfigs {
                  browser {
                    githubWeb {
                      repoUrl('https://github.com/foo/bar')
                    }
                  }
                  gitTool("github")
                  userRemoteConfig {
                    credentialsId("github-creds")
                    name("")
                    refspec("")
                    url("git@github.com:foo/bar.git")
                  }
                }
                branches {
                  branchSpec { name("main") }
                }
              }
            }
            scriptPath("Jenkinsfile.seed")
          }
        }
        properties {
          pipelineTriggers {
            triggers {
              cron { spec('@daily') }
              githubPush()
            }
          }
        }
      }

using a Jenkinsfile to again call the Job DSL:

pipeline {
    agent any

    options {
        timestamps ()
        disableConcurrentBuilds()
    }

    stages {
        stage('Clean') {
            steps {
                deleteDir()
            }
        }

        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Job DSL') {
            steps {
                jobDsl(
                    targets: """
                        jobs/*.groovy
                        views/*.groovy
                    """
                )
            }
        }
    }
}

and create all the jobs/views from that repo (each of which is another Jenkinsfile).

This should allow you to recreate your Jenkins instance, without any manual fiddling; and provide an audit trail of any changes.

Jenkins as Code

Graham HayLeave a comment

Jenkins has come a long way, in the past few years. You can now run it as a docker image:

docker run --rm -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home --name jenkins jenkins/jenkins:lts-jdk11

Or bake your own image, to pre-install plugins:

FROM jenkins/jenkins:lts-jdk11

COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt

providing a list of plugins

antisamy-markup-formatter:latest
build-discarder:latest
configuration-as-code:latest
copyartifact:latest
credentials-binding:latest
envinject:latest
ghprb:latest
git:latest
github:latest
job-dsl:latest
matrix-auth:latest
nodejs:latest
timestamper:latest
workflow-aggregator:latest
ws-cleanup:latest

and now you can even configure those plugins using CasC:

docker run --rm -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home -e CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs -v $PWD/casc_configs:/var/jenkins_home/casc_configs --name jenkins my-jenkins

Jobs that create jobs

Graham HayLeave a comment

Over the last few years, there has been a push for more “* as code” with Jenkins configuration. You can now specify job config using a Jenkinsfile, allowing auditing and code reviews, as well as a backup.

Combined with the Job DSL plugin, this makes it possible to create a seed job (using another Jenkinsfile, naturally) that creates all the jobs for a specific project.

pipeline {
    agent any

    options {
        timestamps ()
    }

    stages {
        stage('Clean') {
            steps {
                deleteDir()
            }
        }

        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Job DSL') {
            steps {
                jobDsl targets: ['jobs/*.groovy', 'views/*.groovy'].join('\n')
            }
        }
    }
}

This will run all the groovy scripts in the jobs & views folders in this repo (once you’ve approved them).

For example:

pipelineJob("foo-main") {
    definition {
        cpsScm{
            scm {
                git {
                    remote {
                        github("examplecorp/foo", "ssh")
                    }
                    branch("main")
                }
            }
            scriptPath("Jenkinsfile")
        }
    }
    properties {
        githubProjectUrl('https://github.com/examplecorp/foo')
        pipelineTriggers {
            triggers {
                cron {
                     spec('@daily')
                }
                githubPush()
            }
        }
    }
}

And a view, to put it in:

listView('foo') {
    description('')

    jobs {
        regex('foo-.*')
    }

    columns {
        status()
        weather()
        name()
        lastSuccess()
        lastFailure()
        lastDuration()
        buildButton()
    }
}

Jenkins and oauth2_proxy

Graham HayLeave a comment

We hide Jenkins behind bitly’s oauth2_proxy, to control access using our Google accounts. After recently upgrading to Debian Jessie (amongst other things), we found that enabling security on Jenkins (using the Reverse Proxy Auth plugin) resulted in an error:

java.lang.NullPointerException
	at org.jenkinsci.plugins.reverse_proxy_auth.ReverseProxySecurityRealm$1.doFilter(ReverseProxySecurityRealm.java:435)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
...

Following the stack trace, we find ourselves here. It’s pretty obvious that the NRE is caused by u being null, but the real question is why we are in that if block at all.

It turns out that at some point the oauth proxy started sending a basic auth header, as well as the X-Forwarded ones we need. This makes the Jenkins plugin sad, when it tries to look up the user.

Unfortunately, there is currently no way to have one without the other, which is an issue for other upstream applications. Hopefully at some point that flag will be added, but until then I’ve simply deleted the offending line.

Jenkins + SSH keys

Graham HayLeave a comment

Jenkins makes it very easy to manage SSH keys. You can use the Credentials plugin to store the key, and then the SSH Agent plugin to seamlessly expose it to your build.

The downside is that now everyone with access to Jenkins has access to that key. It’s possible to use roles to restrict access through the web UI, but in our case it’s useful to allow access to the machine Jenkins is running on (for debugging purposes). And Jenkins itself has r+w privileges, so it’s all but impossible to prevent reading that file.

When the key is used for deploying to production, that’s a problem. Access to the key itself is actually useless, as it’s passphrase protected, but using the solution described above means the passphrase is stored in a credentials.xml file in $JENKINS_HOME. The file is encrypted, but reversing that is trivial.

It would be handy if the SSH Agent plugin allowed prompting for the passphrase before running a build, but that doesn’t appear to be a thing. It is possible however, to use the Parameterized Build plugin to emulate that.

This means you need to start ssh-agent yourself, and due to the fact that ssh-add doesn’t play nice with stdin, there’s some hoop jumping involved. The easiest method seems to involve using expect:

#!/bin/bash

expect << EOF
  spawn ssh-add $1
  expect "Enter passphrase"
  send "$SSH_PASSPHRASE\r"
  expect eof
EOF

Then, assuming you added a build parameter named SSH_PASSPHRASE, you can use this script after launching ssh-agent and before you need the ssh key:

eval `ssh-agent`
./ssh-add-pass ./key_file
./run_playbook