TeamCity build steps are not the way forward

I have a lot of love for TeamCity, and one of the many things that makes it easy to get a build configuration up and running is its concept of “build steps“.

Recently, however, I’ve started to worry about our use (or abuse) of them. My main concern is the fact that it can make it very hard to reproduce a failing build (works on my machine!).

There’s always going to be some differences between the build running on the CI server, and my local build (hardware, installed software, timing, etc). But I’d like to reduce them to the bare minimum. It’s bad enough that I’m running a sh*tty laptop with XP, while the build agent is on 64 bit Server 2008, without having a completely different build process.

I was never a fan of MSBuild, but at least having a build script in with the sauce meant that you could run a local build before checking in.

Some of our build configs now have up to 8 build steps (StyleCop, NuGet install & update, build, test, source index, NuGet pack & publish, etc), any of which could cause a build to fail without being easily reproduced locally. Which encourages people to just check in, and hope for the best. And the slow feedback cycle when trying to fix an issue is a real gumption trap.

So, what’s the alternative? I couldn’t face going back to an xml based build tool (e.g. NAnt or MSBuild), but a scripting language like Ruby (Rake) or Powershell is ideal. In our case, I’d like to replace the steps with a build.ps1 (and some shared modules for common tasks).

Using NuGet 2.0 with TeamCity

We use the NuGet package restore functionality as part of our build. None of the packages are under source control, they are downloaded (from an internal subset of the public feed) before compilation.

Unfortunately, there is a slight shortcoming in some older versions of NuGet that results in far more requests being made than are strictly necessary. This was fixed in version 2.0, but that won’t be supported in TeamCity until 7.1 is released (currently EAP only).

For now, you can choose “custom” as your NuGet version:

Custom NuGet build step

and provide the path to NuGet.exe.

We saw a drastic improvement in build times (from 15-20 mins, down to under 5).

TeamCity publishes broken NuGet packages

We use TeamCity (7.0.3) both as a CI server, and a private NuGet feed. I have a lot of love for TC, but I currently have one major concern:

If my tests fail, TC still publishes the NuGet packages.

Not cool.

I raised this is an issue with JetBrains, and was told it was a feature request. And, that as a workaround, I could split my builds in two and separate the publish step from the rest of the build.

As far as I’m concerned, this is a showstopper bug. It introduces broken behaviour out of the box, which you won’t find out about until you realise a broken package is in use downstream. If JetBrains expects people to use TeamCity as a NuGet feed, this needs to be fixed.

(There’s also no (easy) way to remove packages from the feed, but I’m less concerned about that :) )

Source indexing with PowerShell (and TeamCity)

Source indexing is definitely a “best practice”, when developing libraries that will be referenced as binaries.

Getting it working as part of a CI build can be a bit fiddly though.

There are a few pre-requisites for the build agent:

  1. Perl (>= 5.6): Strawberry Perl portable, for example
  2. SrcSrv: Part of the Debugging Tools for Windows
  3. SVN: command line e.g. Win32SVN (zip install)

All the above tools can be xcopy installed, but feel free to use an msi.

To index your PDBs, you need to run svnindex. Using TeamCity, you can add a PowerShell build step (in our case, after build & test, and before NuGet packaging):

function srcIndex([string] $project)
{
  & svnindex.cmd /debug /source=$project /symbols="$project\bin"
}

write-host "Updating path"
$env:path = "$env:path;D:\perl\perl\site\bin;D:\perl\perl\bin;D:\perl\c\bin;D:\srcsrv;D:\svn\bin"
write-host "Path: $env:path"
$env:term = "dumb" #strawberry perl portable specific

srcIndex "%system.teamcity.build.checkoutDir%\src\MyProject"

We start by updating the path, to include the location of the necessary exes (you can skip this if you used an installer for the pre-reqs). We then point svnindex at the source, and symbols.

If you want to index all your projects, you can loop over them:

gci "%system.teamcity.build.checkoutDir%\src" -r -i *.csproj | foreach { srcIndex $_.fullname }

The working dir for the build step needs to be the drive root e.g. D:\, as the indexing scripts don’t like relative paths (and you’ll see the dreaded “… zero source files found …”).

Then you just need to enable Source Server support in Visual Studio (uncheck “Just my code”), and luxuriate in full source debugging!

EDIT: Make sure the VCS root is set to checkout on the agent, not the server, as the information required is in the SVN repo.

Build Monkey

Following in the footsteps of the Netflix Chaos Monkey, I created a Build Monkey.

If you point it at a TeamCity instance (with the REST API plugin installed), it will randomly select a build and run it.

The idea is to make sure that all builds get run reasonably regularly, and avoid the unpleasant surprise when one that hasn’t been run for 6 months explodes in your face.

(At the moment it won’t run builds with “Live” in the title e.g. “Deploy Foo to Live” (because I’m a chicken :). But really these should also be run regularly, even if the code hasn’t changed).

It would be better if it was biased towards builds that hadn’t been run recently, but I’m not sure if the TeamCity REST API provides enough data for that.