Piping two commands together in Go

The Go exec package makes it pretty easy to run an OS command and read the output; and there are a few examples of how to pipe the output of one command into another (e.g. lsof | wc -l):

lsof := exec.Command("lsof")
wc := exec.Command("wc", "-l")
outPipe, err := lsof.StdoutPipe()
if err != nil {
    return nil, err
}
lsof.Start()
wc.Stdin = outPipe
out, err := wc.Output()
if err != nil {
    return nil, err
}
return out, nil

There’s a problem with this code though, the pipe will never be closed; if the caller is a long-lived process then it will leak file descriptors.

You don’t normally have to worry about this, as calling Run or Output (instead of Start) will take care of it; but we can’t use those as they’ll close the pipe too early.

The problem is easily solved, either by closing the pipe yourself:

outPipe, err := lsof.StdoutPipe()
defer outPipe.Close()

or by calling Wait after the output from the second command has been received:

out, err := wc.Output()
if err != nil {
    return nil, err
}
err = lsof.Wait()
if err != nil {
    return nil, err
}
return out, nil

A generic json handler in Go

One of the things I really like about Go is the way it encourages you to build up behaviour using composition. For example, I was writing handlers that return a response object as json; it’s very simple to extract the duplicated code into a generic handler:

package main

import (
	"encoding/json"
	"log"
	"net/http"
)

type ResponseGenerator interface {
	GetResponse(r *http.Request) (interface{}, error)
}

type JsonHandler struct {
	log *log.Logger
	rg  ResponseGenerator
}

func (h *JsonHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	response, err := h.rg.GetResponse(r)
	if err != nil {
		h.log.Printf("ERROR: %v\n", err)
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	b, err := json.Marshal(response)
	if err != nil {
		h.log.Printf("ERROR: %v\n", err)
		http.Error(w, "Unexpected error", http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Type", "application/json")
	w.Write(b)
}

Running the dieharder test suite against a Fortuna implementation

I wanted to run the dieharder suite of tests against this implementation of the Fortuna PRNG algorithm. Once I’d worked out how to dump binary output to stdout, it was pretty easy to generate an infinite stream of (hopefully) random data that could be piped into dieharder:

package main

import (
	"crypto/aes"
	"encoding/binary"
	"os"

	"github.com/seehuhn/fortuna"
)

func main() {
	rng, err := fortuna.NewRNG("")
	if err != nil {
		panic("cannot initialise the RNG: " + err.Error())
	}
	defer rng.Close()
	gen := fortuna.NewGenerator(aes.NewCipher)
	for true {
		err = binary.Write(os.Stdout, binary.LittleEndian, gen.PseudoRandomData(16))
		if err != nil {
			panic("binary.Write failed:" + err.Error())
		}
	}
}

(The source can be found here). Below is an example output:

$ go run src/fortuna-dieharder.go | dieharder -a -g 200 -k 2 -Y 1
#=============================================================================#
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |rands/second|   Seed   |
stdin_input_raw|  3.79e+05  |4078583786|
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|     100|0.79585928|  PASSED  
      diehard_operm5|   0|   1000000|     100|0.78748529|  PASSED  
  diehard_rank_32x32|   0|     40000|     100|0.65244868|  PASSED  
    diehard_rank_6x8|   0|    100000|     100|0.15429465|  PASSED  
   diehard_bitstream|   0|   2097152|     100|0.48566800|  PASSED  
        diehard_opso|   0|   2097152|     100|0.35772252|  PASSED  
        diehard_oqso|   0|   2097152|     100|0.47349980|  PASSED  
         diehard_dna|   0|   2097152|     100|0.48349937|  PASSED  
diehard_count_1s_str|   0|    256000|     100|0.95257057|  PASSED  
diehard_count_1s_byt|   0|    256000|     100|0.84437819|  PASSED  
 diehard_parking_lot|   0|     12000|     100|0.31899023|  PASSED  
    diehard_2dsphere|   2|      8000|     100|0.24483205|  PASSED  
    diehard_3dsphere|   3|      4000|     100|0.04900588|  PASSED  
     diehard_squeeze|   0|    100000|     100|0.96659492|  PASSED  
        diehard_sums|   0|       100|     100|0.92162735|  PASSED  
        diehard_runs|   0|    100000|     100|0.29733372|  PASSED  
        diehard_runs|   0|    100000|     100|0.21733297|  PASSED  
       diehard_craps|   0|    200000|     100|0.00620028|  PASSED  
       diehard_craps|   0|    200000|     100|0.77686961|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.41392103|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.94153849|  PASSED  
         sts_monobit|   1|    100000|     100|0.97969393|  PASSED  
            sts_runs|   2|    100000|     100|0.58348717|  PASSED  
          sts_serial|   1|    100000|     100|0.43332780|  PASSED  
          sts_serial|   2|    100000|     100|0.45215572|  PASSED  
          sts_serial|   3|    100000|     100|0.63888152|  PASSED  
          sts_serial|   3|    100000|     100|0.24532215|  PASSED  
          sts_serial|   4|    100000|     100|0.82498873|  PASSED  
          sts_serial|   4|    100000|     100|0.70631562|  PASSED  
          sts_serial|   5|    100000|     100|0.14098307|  PASSED  
          sts_serial|   5|    100000|     100|0.05971648|  PASSED  
          sts_serial|   6|    100000|     100|0.44033615|  PASSED  
          sts_serial|   6|    100000|     100|0.84254288|  PASSED  
          sts_serial|   7|    100000|     100|0.94534057|  PASSED  
          sts_serial|   7|    100000|     100|0.56895078|  PASSED  
          sts_serial|   8|    100000|     100|0.17537892|  PASSED  
          sts_serial|   8|    100000|     100|0.41596536|  PASSED  
          sts_serial|   9|    100000|     100|0.93984064|  PASSED  
          sts_serial|   9|    100000|     100|0.77199673|  PASSED  
          sts_serial|  10|    100000|     100|0.49608967|  PASSED  
          sts_serial|  10|    100000|     100|0.19713502|  PASSED  
          sts_serial|  11|    100000|     100|0.32361987|  PASSED  
          sts_serial|  11|    100000|     100|0.28265068|  PASSED  
          sts_serial|  12|    100000|     100|0.32824861|  PASSED  
          sts_serial|  12|    100000|     100|0.43554391|  PASSED  
          sts_serial|  13|    100000|     100|0.01463804|  PASSED  
          sts_serial|  13|    100000|     100|0.24260554|  PASSED  
          sts_serial|  14|    100000|     100|0.95754708|  PASSED  
          sts_serial|  14|    100000|     100|0.11090928|  PASSED  
          sts_serial|  15|    100000|     100|0.62735234|  PASSED  
          sts_serial|  15|    100000|     100|0.97145571|  PASSED  
          sts_serial|  16|    100000|     100|0.45211316|  PASSED  
          sts_serial|  16|    100000|     100|0.60723389|  PASSED  
         rgb_bitdist|   1|    100000|     100|0.36658874|  PASSED  
         rgb_bitdist|   2|    100000|     100|0.99411631|  PASSED  
         rgb_bitdist|   3|    100000|     100|0.06557851|  PASSED  
         rgb_bitdist|   4|    100000|     100|0.09920095|  PASSED  
         rgb_bitdist|   5|    100000|     100|0.69241623|  PASSED  
         rgb_bitdist|   6|    100000|     100|0.34436231|  PASSED  
         rgb_bitdist|   7|    100000|     100|0.79110397|  PASSED  
         rgb_bitdist|   8|    100000|     100|0.15949927|  PASSED  
         rgb_bitdist|   9|    100000|     100|0.82775041|  PASSED  
         rgb_bitdist|  10|    100000|     100|0.61202477|  PASSED  
         rgb_bitdist|  11|    100000|     100|0.88799469|  PASSED  
         rgb_bitdist|  12|    100000|     100|0.96488193|  PASSED  
rgb_minimum_distance|   2|     10000|    1000|0.35186805|  PASSED  
rgb_minimum_distance|   3|     10000|    1000|0.21343396|  PASSED  
rgb_minimum_distance|   4|     10000|    1000|0.18466470|  PASSED  
rgb_minimum_distance|   5|     10000|    1000|0.49211455|  PASSED  
    rgb_permutations|   2|    100000|     100|0.93013365|  PASSED  
    rgb_permutations|   3|    100000|     100|0.24389986|  PASSED  
    rgb_permutations|   4|    100000|     100|0.10744314|  PASSED  
    rgb_permutations|   5|    100000|     100|0.86806297|  PASSED  
      rgb_lagged_sum|   0|   1000000|     100|0.57089479|  PASSED  
      rgb_lagged_sum|   1|   1000000|     100|0.63991216|  PASSED  
      rgb_lagged_sum|   2|   1000000|     100|0.89554236|  PASSED  
      rgb_lagged_sum|   3|   1000000|     100|0.94219831|  PASSED  
      rgb_lagged_sum|   4|   1000000|     100|0.76276793|  PASSED  
      rgb_lagged_sum|   5|   1000000|     100|0.92606410|  PASSED  
      rgb_lagged_sum|   6|   1000000|     100|0.93189843|  PASSED  
      rgb_lagged_sum|   7|   1000000|     100|0.98320319|  PASSED  
      rgb_lagged_sum|   8|   1000000|     100|0.92024295|  PASSED  
      rgb_lagged_sum|   9|   1000000|     100|0.13888672|  PASSED  
      rgb_lagged_sum|  10|   1000000|     100|0.76576110|  PASSED  
      rgb_lagged_sum|  11|   1000000|     100|0.82383503|  PASSED  
      rgb_lagged_sum|  12|   1000000|     100|0.98684700|  PASSED  
      rgb_lagged_sum|  13|   1000000|     100|0.26130464|  PASSED  
      rgb_lagged_sum|  14|   1000000|     100|0.26742272|  PASSED  
      rgb_lagged_sum|  15|   1000000|     100|0.76937245|  PASSED  
      rgb_lagged_sum|  16|   1000000|     100|0.05400234|  PASSED  
      rgb_lagged_sum|  17|   1000000|     100|0.11081369|  PASSED  
      rgb_lagged_sum|  18|   1000000|     100|0.89407701|  PASSED  
      rgb_lagged_sum|  19|   1000000|     100|0.83792478|  PASSED  
      rgb_lagged_sum|  20|   1000000|     100|0.61427436|  PASSED  
      rgb_lagged_sum|  21|   1000000|     100|0.66549271|  PASSED  
      rgb_lagged_sum|  22|   1000000|     100|0.18433339|  PASSED  
      rgb_lagged_sum|  23|   1000000|     100|0.63460123|  PASSED  
      rgb_lagged_sum|  24|   1000000|     100|0.99377481|  PASSED  
      rgb_lagged_sum|  25|   1000000|     100|0.48356785|  PASSED  
      rgb_lagged_sum|  26|   1000000|     100|0.49883205|  PASSED  
      rgb_lagged_sum|  27|   1000000|     100|0.04619725|  PASSED  
      rgb_lagged_sum|  28|   1000000|     100|0.64940352|  PASSED  
      rgb_lagged_sum|  29|   1000000|     100|0.96496460|  PASSED  
      rgb_lagged_sum|  30|   1000000|     100|0.70586167|  PASSED  
      rgb_lagged_sum|  31|   1000000|     100|0.58423701|  PASSED  
      rgb_lagged_sum|  32|   1000000|     100|0.46024310|  PASSED  
     rgb_kstest_test|   0|     10000|    1000|0.09525747|  PASSED  
     dab_bytedistrib|   0|  51200000|       1|0.72957242|  PASSED  
             dab_dct| 256|     50000|       1|0.67061200|  PASSED  
Preparing to run test 207.  ntuple = 0
        dab_filltree|  32|  15000000|       1|0.44986941|  PASSED  
        dab_filltree|  32|  15000000|       1|0.55626134|  PASSED  
Preparing to run test 208.  ntuple = 0
       dab_filltree2|   0|   5000000|       1|0.06208234|  PASSED  
       dab_filltree2|   1|   5000000|       1|0.65494424|  PASSED  
Preparing to run test 209.  ntuple = 0
        dab_monobit2|  12|  65000000|       1|0.68823690|  PASSED  

However, bear in mind that:

Most of the tests in Diehard return a p-value, which should be uniform on [0,1) if the input file contains truly independent random bits. Those p-values are obtained by p=F(X), where F is the assumed distribution of the sample random variable X—often normal. But that assumed F is just an asymptotic approximation, for which the fit will be worst in the tails. Thus you should not be surprised with occasional p-values near 0 or 1, such as .0012 or .9983.

When a bit stream really FAILS BIG, you will get p’s of 0 or 1 to six or more places. By all means, do not, as a Statistician might, think that a p .975 means that the RNG has “failed the test at the .05 level”. Such p’s happen among the hundreds that Diehard produces, even with good RNG’s. So keep in mind that “p happens”.

Printing a raw binary to stdout with Go

This is probably just me, but I found it surprisingly hard to output some raw binary data to stdout using Go (obviously it’s easy when tha knows how!). I [insert search engine here]ed it pretty hard, but couldn’t find any examples.

Using any of the methods from the fmt package resulted in a representation being printed, rather than the raw binary:

fmt.Println(gen.PseudoRandomData(16))

$go run src/print-binary1.go 
[19 234 189 238 126 50 65 77 46 130 105 26 50 12 217 195]

Then I looked at the binary package, but the only example was writing to a []byte buffer. Eventually, I found a breadcrumb in one of the mailing lists that led me to the solution:

binary.Write(os.Stdout, binary.LittleEndian, gen.PseudoRandomData(16))