Zero downtime deployments with node cluster

The easiest way to do zero downtime deployments is using multiple nodes behind a load balancer. Once removed from the rotation, you can fiddle with them to your heart’s content.

If you only have one box to play with, things aren’t so simple. One option is to push the complexity up to whatever you’re using to orchestrate deployments. You could do something like a blue/green deployment, with two full sets of processes behind nginx as a load balancer; but this felt liable to be very fragile.

I next started looking at a process manager, like pm2; but it seemed to offer far too many features I didn’t need, and didn’t play that well with systemd. It was inspiration though, for just going direct to the nodejs cluster API. It’s been available for some time, and is now marked as “stable”.

It allows you to run multiple node processes that share a port, which is also useful if you are running on hardware with multiple CPUs. Using the cluster API allows us to recycle the processes when the code has changed, without dropping any requests:

var cluster = require('cluster');

module.exports = function(start) {
    var env = process.env.NODE_ENV || 'local';
    var cwd = process.cwd();

    var numWorkers = process.env.NUM_WORKERS || 1;

    if (cluster.isMaster) {

        cluster.on('exit', function(worker, code, signal) {
            // one for all, let systemd deal with it
            if (code !== 0) {

        process.on('SIGHUP', function() {
            // need to chdir, if the old directory was deleted
            var oldWorkers = Object.keys(cluster.workers);
            // wait until at least one new worker is listening
            // before terminating the old workers
            cluster.once('listening', function(worker, address) {
    } else {

    function fork() {
        for (var i = 0; i < numWorkers; i++) {

    function kill(workers) {
        if (workers.length) {
            var id = workers.pop();
            var worker = cluster.workers[id];

During a deployment, we simply replace the old code and send a signal to the “master” process (kill -HUP $PID); which causes it to spin up X new workers. As soon as one of those is ready and listening, it terminates the old workers.

Adding a domain user to a local group using PowerShell

Following on from the fun of giving write permissions on a folder to a user, today’s installment covers adding a domain user to a local group.

Specifically, the group “Performance Log Users”, which allows a process to use (rather than create) perf counters.

function Add-UserToPerformanceLogUsersGroup($user, $session) {  
  Invoke-Command -Args $user -Session $session -ErrorAction Stop -ScriptBlock {
    $groupName = "Performance Log Users"
    $group = [ADSI]("WinNT://$env:COMPUTERNAME/$groupName,group")
    # check if user is already a member
    $members = @($group.psbase.Invoke("Members"))
    $matches = $members | where { $_.GetType().InvokeMember("Name", 'GetProperty', $null, $_, $null) -eq $user.split("\")[1] }
    if ($matches -eq $null) {
      Write-Host "Adding $user to $groupName group"   
      $user = $user.replace("\", "/")

Caveat: the user specified is assumed to be a fully qualified DOMAIN\User, hence the unpleasant string manipulation.

Unable to generate a temporary class (result=1)

A recently deployed web service (IIS on Server 2008) was producing a YSOD with the error:

Unable to generate a temporary class (result=1)

Some cursory duckduckgo-ing suggested that the problem was a lack of permissions, for the user that IIS was running as, on the C:\Windows\Temp folder.

I compared the rights for that folder on the broken server with a working one, and the (local) IIS_IUSRS group had “special permissions”. Specifically, the right to “List folder / read data”.

Once that was set up, IIS was back in business. I presume this is something normally set up during IIS install, which either went wrong or was later corrupted.

Multiple config transforms (at build time)

The web.config transforms always looked like a really nice way to remove duplication. I had a few concerns though: they didn’t work with app.configs, and I don’t want to use MS Deploy.

There are a few solutions available for the first problem, e.g. SlowCheetah. But it turns out there’s a much simpler option available (as long as you’re not afraid of butchering a csproj file!).

Adding a transform is just a matter of a new content node:

<None Include="App.config" />
<None Include="App.DEV.config">

Then, to generate the transform at build time (rather than “publishing”) add the following at the end of your csproj file:

<UsingTask TaskName="TransformXml" AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll" />
<Target Name="AfterCompile">
  <TransformXml Source="App.config" Transform="App.DEV.config" Destination="$(OutDir)\App.DEV.config" StackTrace="true" />

Now, all your transformed config files will be in the bin dir after building, ready for deployment!

If you haven’t got VS installed on the build agent, you might need to copy the required DLLs. (Also, make sure you have SP1 installed, there’s a nasty bug with excess line breaks).

(All credit for this belongs to a co-worker of mine, but he doesn’t have a blog and it seems a shame to let it go to waste. Unless he found it on the internet, in which case “nul point”).

PowerShell equivalent of Capistrano’s “run”

Automating deployments using PowerShell & WinRM is, by turns, both awesome and deeply frustrating.

One of the main stumbling blocks, for me anyway, is which side of the machine boundary the code should be evaluated on.

When using Invoke-Command with a script block, any variables inside the block are evaluated remotely:

$foo = "foo"
Invoke-Command -ScriptBlock {
    Write-Host "Foo: $foo"

If you want to use a local variable on the remote machine, you need to force the evaluation earlier. I ended up with something inspired by Capistrano’s run action.

function run([string] $command, $session)
  $runBlock = [ScriptBlock]::Create($command)
    Invoke-Command -Session $session -ScriptBlock $runBlock -ErrorAction Stop
    exit 1

$foo = "foo"
run "write-host `"Foo: $foo`""

Alternatively, you can pass the args to Invoke-Command explicitly:

Invoke-Command -Args $foo, $bar -Session $session -ErrorAction Stop -ScriptBlock {
  param($foo, $bar)
  Write-Host "Foo: $foo"
  Write-Host "Bar: $bar"