Docker provides a simple way to build a golang project, without needing to install the toolchain:
docker run -v $PWD:/go/bin golang:stretch go get -u github.com/pressly/goose/cmd/goose
The built binary should appear in the pwd.
Golang
Connecting to postgres using domain sockets and libpq
We recently started using Goose to run our migrations, and I wanted to connect to the Postgres instance on a local machine using a domain socket. The documentation states:
host
Name of host to connect to. If this begins with a slash, it specifies Unix-domain communication rather than TCP/IP communication; the value is the name of the directory in which the socket file is stored. The default behavior when host is not specified is to connect to a Unix-domain socket in /tmp (or whatever socket directory was specified when PostgreSQL was built). On machines without Unix-domain sockets, the default is to connect to localhost.
So I assumed that specifying a connection string without a host would “just work”:
db, err := sql.Open("postgres", "dbname=foo")
This doesn’t return an error, but any attempt to use the connection resulted in a “bad connection” failure. For some reason I needed to specify the path to the socket (which wasn’t in /tmp on the Debian instance I was using!):
db, err := sql.Open("postgres", "host=/run/postgresql dbname=foo sslmode=disable")
Producer-consumer using Go channels
I needed to call a web service a specific number of times, and add the results to a file (order being unimportant). It might be possible to simply spawn a goroutine for every request, but a more elegant solution is to use the producer-consumer pattern:
func main() {
jobs := make(chan bool)
results := make(chan string)
done := make(chan bool)
numberOfWorkers:= 5
numberOfJobs:= 1000
for i := 0; i < numberOfWorkers; i++ {
go worker(jobs, results, done)
}
go func() {
for i := 0; i < numberOfJobs; i++ {
jobs <- true
}
}()
go func() {
count := 0
for {
result := <-results
println(result)
count++
if count >= numberOfJobs {
done <- true
return
}
}
}()
<-done
}
func worker(jobs chan bool, result chan string, done chan bool) {
for {
select {
case <-jobs:
res, err := getResult()
if err != nil {
panic(err)
}
results <- res
case <-done:
return
}
}
}
func getResult() (string, error) {
resp, err := http.Get("http://localhost/foo")
if err != nil {
return 0, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return 0, err
}
return body, nil
}
This solution uses 3 channels: one to specify the work still remaining (a bool in this case, but this channel could include data to vary the request), one to return results from the workers; and one to signify that work was complete.
This turned out to be a more interesting problem than it looked. My first attempt was using a WaitGroup, but I couldn’t get that to work. And debugging revealed an interesting gap in my mental model of how channels work: I hadn’t fully internalised the fact that pushing a message onto a go channel will block unless someone is ready to consume it. Hence the need to push work onto the jobs channel in a separate goroutine. I had it in my head that they were more like an Erlang/F# mailbox, despite the fact I knew that buffered channels were a thing.
The error handling could also be improved, currently if any request fails it just panics.
Piping two commands together in Go
The Go exec package makes it pretty easy to run an OS command and read the output; and there are a few examples of how to pipe the output of one command into another (e.g. lsof | wc -l):
lsof := exec.Command("lsof")
wc := exec.Command("wc", "-l")
outPipe, err := lsof.StdoutPipe()
if err != nil {
return nil, err
}
lsof.Start()
wc.Stdin = outPipe
out, err := wc.Output()
if err != nil {
return nil, err
}
return out, nil
There’s a problem with this code though, the pipe will never be closed; if the caller is a long-lived process then it will leak file descriptors.
You don’t normally have to worry about this, as calling Run or Output (instead of Start) will take care of it; but we can’t use those as they’ll close the pipe too early.
The problem is easily solved, either by closing the pipe yourself:
outPipe, err := lsof.StdoutPipe() defer outPipe.Close()
or by calling Wait after the output from the second command has been received:
out, err := wc.Output()
if err != nil {
return nil, err
}
err = lsof.Wait()
if err != nil {
return nil, err
}
return out, nil
A generic json handler in Go
One of the things I really like about Go is the way it encourages you to build up behaviour using composition. For example, I was writing handlers that return a response object as json; it’s very simple to extract the duplicated code into a generic handler:
package main
import (
"encoding/json"
"log"
"net/http"
)
type ResponseGenerator interface {
GetResponse(r *http.Request) (interface{}, error)
}
type JsonHandler struct {
log *log.Logger
rg ResponseGenerator
}
func (h *JsonHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
response, err := h.rg.GetResponse(r)
if err != nil {
h.log.Printf("ERROR: %v\n", err)
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
b, err := json.Marshal(response)
if err != nil {
h.log.Printf("ERROR: %v\n", err)
http.Error(w, "Unexpected error", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
w.Write(b)
}
Running the dieharder test suite against a Fortuna implementation
I wanted to run the dieharder suite of tests against this implementation of the Fortuna PRNG algorithm. Once I’d worked out how to dump binary output to stdout, it was pretty easy to generate an infinite stream of (hopefully) random data that could be piped into dieharder:
package main
import (
"crypto/aes"
"encoding/binary"
"os"
"github.com/seehuhn/fortuna"
)
func main() {
rng, err := fortuna.NewRNG("")
if err != nil {
panic("cannot initialise the RNG: " + err.Error())
}
defer rng.Close()
gen := fortuna.NewGenerator(aes.NewCipher)
for true {
err = binary.Write(os.Stdout, binary.LittleEndian, gen.PseudoRandomData(16))
if err != nil {
panic("binary.Write failed:" + err.Error())
}
}
}
(The source can be found here). Below is an example output:
$ go run src/fortuna-dieharder.go | dieharder -a -g 200 -k 2 -Y 1
#=============================================================================#
# dieharder version 3.31.1 Copyright 2003 Robert G. Brown #
#=============================================================================#
rng_name |rands/second| Seed |
stdin_input_raw| 3.79e+05 |4078583786|
#=============================================================================#
test_name |ntup| tsamples |psamples| p-value |Assessment
#=============================================================================#
diehard_birthdays| 0| 100| 100|0.79585928| PASSED
diehard_operm5| 0| 1000000| 100|0.78748529| PASSED
diehard_rank_32x32| 0| 40000| 100|0.65244868| PASSED
diehard_rank_6x8| 0| 100000| 100|0.15429465| PASSED
diehard_bitstream| 0| 2097152| 100|0.48566800| PASSED
diehard_opso| 0| 2097152| 100|0.35772252| PASSED
diehard_oqso| 0| 2097152| 100|0.47349980| PASSED
diehard_dna| 0| 2097152| 100|0.48349937| PASSED
diehard_count_1s_str| 0| 256000| 100|0.95257057| PASSED
diehard_count_1s_byt| 0| 256000| 100|0.84437819| PASSED
diehard_parking_lot| 0| 12000| 100|0.31899023| PASSED
diehard_2dsphere| 2| 8000| 100|0.24483205| PASSED
diehard_3dsphere| 3| 4000| 100|0.04900588| PASSED
diehard_squeeze| 0| 100000| 100|0.96659492| PASSED
diehard_sums| 0| 100| 100|0.92162735| PASSED
diehard_runs| 0| 100000| 100|0.29733372| PASSED
diehard_runs| 0| 100000| 100|0.21733297| PASSED
diehard_craps| 0| 200000| 100|0.00620028| PASSED
diehard_craps| 0| 200000| 100|0.77686961| PASSED
marsaglia_tsang_gcd| 0| 10000000| 100|0.41392103| PASSED
marsaglia_tsang_gcd| 0| 10000000| 100|0.94153849| PASSED
sts_monobit| 1| 100000| 100|0.97969393| PASSED
sts_runs| 2| 100000| 100|0.58348717| PASSED
sts_serial| 1| 100000| 100|0.43332780| PASSED
sts_serial| 2| 100000| 100|0.45215572| PASSED
sts_serial| 3| 100000| 100|0.63888152| PASSED
sts_serial| 3| 100000| 100|0.24532215| PASSED
sts_serial| 4| 100000| 100|0.82498873| PASSED
sts_serial| 4| 100000| 100|0.70631562| PASSED
sts_serial| 5| 100000| 100|0.14098307| PASSED
sts_serial| 5| 100000| 100|0.05971648| PASSED
sts_serial| 6| 100000| 100|0.44033615| PASSED
sts_serial| 6| 100000| 100|0.84254288| PASSED
sts_serial| 7| 100000| 100|0.94534057| PASSED
sts_serial| 7| 100000| 100|0.56895078| PASSED
sts_serial| 8| 100000| 100|0.17537892| PASSED
sts_serial| 8| 100000| 100|0.41596536| PASSED
sts_serial| 9| 100000| 100|0.93984064| PASSED
sts_serial| 9| 100000| 100|0.77199673| PASSED
sts_serial| 10| 100000| 100|0.49608967| PASSED
sts_serial| 10| 100000| 100|0.19713502| PASSED
sts_serial| 11| 100000| 100|0.32361987| PASSED
sts_serial| 11| 100000| 100|0.28265068| PASSED
sts_serial| 12| 100000| 100|0.32824861| PASSED
sts_serial| 12| 100000| 100|0.43554391| PASSED
sts_serial| 13| 100000| 100|0.01463804| PASSED
sts_serial| 13| 100000| 100|0.24260554| PASSED
sts_serial| 14| 100000| 100|0.95754708| PASSED
sts_serial| 14| 100000| 100|0.11090928| PASSED
sts_serial| 15| 100000| 100|0.62735234| PASSED
sts_serial| 15| 100000| 100|0.97145571| PASSED
sts_serial| 16| 100000| 100|0.45211316| PASSED
sts_serial| 16| 100000| 100|0.60723389| PASSED
rgb_bitdist| 1| 100000| 100|0.36658874| PASSED
rgb_bitdist| 2| 100000| 100|0.99411631| PASSED
rgb_bitdist| 3| 100000| 100|0.06557851| PASSED
rgb_bitdist| 4| 100000| 100|0.09920095| PASSED
rgb_bitdist| 5| 100000| 100|0.69241623| PASSED
rgb_bitdist| 6| 100000| 100|0.34436231| PASSED
rgb_bitdist| 7| 100000| 100|0.79110397| PASSED
rgb_bitdist| 8| 100000| 100|0.15949927| PASSED
rgb_bitdist| 9| 100000| 100|0.82775041| PASSED
rgb_bitdist| 10| 100000| 100|0.61202477| PASSED
rgb_bitdist| 11| 100000| 100|0.88799469| PASSED
rgb_bitdist| 12| 100000| 100|0.96488193| PASSED
rgb_minimum_distance| 2| 10000| 1000|0.35186805| PASSED
rgb_minimum_distance| 3| 10000| 1000|0.21343396| PASSED
rgb_minimum_distance| 4| 10000| 1000|0.18466470| PASSED
rgb_minimum_distance| 5| 10000| 1000|0.49211455| PASSED
rgb_permutations| 2| 100000| 100|0.93013365| PASSED
rgb_permutations| 3| 100000| 100|0.24389986| PASSED
rgb_permutations| 4| 100000| 100|0.10744314| PASSED
rgb_permutations| 5| 100000| 100|0.86806297| PASSED
rgb_lagged_sum| 0| 1000000| 100|0.57089479| PASSED
rgb_lagged_sum| 1| 1000000| 100|0.63991216| PASSED
rgb_lagged_sum| 2| 1000000| 100|0.89554236| PASSED
rgb_lagged_sum| 3| 1000000| 100|0.94219831| PASSED
rgb_lagged_sum| 4| 1000000| 100|0.76276793| PASSED
rgb_lagged_sum| 5| 1000000| 100|0.92606410| PASSED
rgb_lagged_sum| 6| 1000000| 100|0.93189843| PASSED
rgb_lagged_sum| 7| 1000000| 100|0.98320319| PASSED
rgb_lagged_sum| 8| 1000000| 100|0.92024295| PASSED
rgb_lagged_sum| 9| 1000000| 100|0.13888672| PASSED
rgb_lagged_sum| 10| 1000000| 100|0.76576110| PASSED
rgb_lagged_sum| 11| 1000000| 100|0.82383503| PASSED
rgb_lagged_sum| 12| 1000000| 100|0.98684700| PASSED
rgb_lagged_sum| 13| 1000000| 100|0.26130464| PASSED
rgb_lagged_sum| 14| 1000000| 100|0.26742272| PASSED
rgb_lagged_sum| 15| 1000000| 100|0.76937245| PASSED
rgb_lagged_sum| 16| 1000000| 100|0.05400234| PASSED
rgb_lagged_sum| 17| 1000000| 100|0.11081369| PASSED
rgb_lagged_sum| 18| 1000000| 100|0.89407701| PASSED
rgb_lagged_sum| 19| 1000000| 100|0.83792478| PASSED
rgb_lagged_sum| 20| 1000000| 100|0.61427436| PASSED
rgb_lagged_sum| 21| 1000000| 100|0.66549271| PASSED
rgb_lagged_sum| 22| 1000000| 100|0.18433339| PASSED
rgb_lagged_sum| 23| 1000000| 100|0.63460123| PASSED
rgb_lagged_sum| 24| 1000000| 100|0.99377481| PASSED
rgb_lagged_sum| 25| 1000000| 100|0.48356785| PASSED
rgb_lagged_sum| 26| 1000000| 100|0.49883205| PASSED
rgb_lagged_sum| 27| 1000000| 100|0.04619725| PASSED
rgb_lagged_sum| 28| 1000000| 100|0.64940352| PASSED
rgb_lagged_sum| 29| 1000000| 100|0.96496460| PASSED
rgb_lagged_sum| 30| 1000000| 100|0.70586167| PASSED
rgb_lagged_sum| 31| 1000000| 100|0.58423701| PASSED
rgb_lagged_sum| 32| 1000000| 100|0.46024310| PASSED
rgb_kstest_test| 0| 10000| 1000|0.09525747| PASSED
dab_bytedistrib| 0| 51200000| 1|0.72957242| PASSED
dab_dct| 256| 50000| 1|0.67061200| PASSED
Preparing to run test 207. ntuple = 0
dab_filltree| 32| 15000000| 1|0.44986941| PASSED
dab_filltree| 32| 15000000| 1|0.55626134| PASSED
Preparing to run test 208. ntuple = 0
dab_filltree2| 0| 5000000| 1|0.06208234| PASSED
dab_filltree2| 1| 5000000| 1|0.65494424| PASSED
Preparing to run test 209. ntuple = 0
dab_monobit2| 12| 65000000| 1|0.68823690| PASSED
However, bear in mind that:
Most of the tests in Diehard return a p-value, which should be uniform on [0,1) if the input file contains truly independent random bits. Those p-values are obtained by p=F(X), where F is the assumed distribution of the sample random variable X—often normal. But that assumed F is just an asymptotic approximation, for which the fit will be worst in the tails. Thus you should not be surprised with occasional p-values near 0 or 1, such as .0012 or .9983.
When a bit stream really FAILS BIG, you will get p’s of 0 or 1 to six or more places. By all means, do not, as a Statistician might, think that a p .975 means that the RNG has “failed the test at the .05 level”. Such p’s happen among the hundreds that Diehard produces, even with good RNG’s. So keep in mind that “p happens”.
Printing a raw binary to stdout with Go
This is probably just me, but I found it surprisingly hard to output some raw binary data to stdout using Go (obviously it’s easy when tha knows how!). I [insert search engine here]ed it pretty hard, but couldn’t find any examples.
Using any of the methods from the fmt package resulted in a representation being printed, rather than the raw binary:
fmt.Println(gen.PseudoRandomData(16)) $go run src/print-binary1.go [19 234 189 238 126 50 65 77 46 130 105 26 50 12 217 195]
Then I looked at the binary package, but the only example was writing to a []byte buffer. Eventually, I found a breadcrumb in one of the mailing lists that led me to the solution:
binary.Write(os.Stdout, binary.LittleEndian, gen.PseudoRandomData(16))
Look on my works, ye Mighty, and despair! 