Running the dieharder test suite against a Fortuna implementation

I wanted to run the dieharder suite of tests against this implementation of the Fortuna PRNG algorithm. Once I’d worked out how to dump binary output to stdout, it was pretty easy to generate an infinite stream of (hopefully) random data that could be piped into dieharder:

package main

import (
	"crypto/aes"
	"encoding/binary"
	"os"

	"github.com/seehuhn/fortuna"
)

func main() {
	rng, err := fortuna.NewRNG("")
	if err != nil {
		panic("cannot initialise the RNG: " + err.Error())
	}
	defer rng.Close()
	gen := fortuna.NewGenerator(aes.NewCipher)
	for true {
		err = binary.Write(os.Stdout, binary.LittleEndian, gen.PseudoRandomData(16))
		if err != nil {
			panic("binary.Write failed:" + err.Error())
		}
	}
}

(The source can be found here). Below is an example output:

$ go run src/fortuna-dieharder.go | dieharder -a -g 200 -k 2 -Y 1
#=============================================================================#
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |rands/second|   Seed   |
stdin_input_raw|  3.79e+05  |4078583786|
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|     100|0.79585928|  PASSED  
      diehard_operm5|   0|   1000000|     100|0.78748529|  PASSED  
  diehard_rank_32x32|   0|     40000|     100|0.65244868|  PASSED  
    diehard_rank_6x8|   0|    100000|     100|0.15429465|  PASSED  
   diehard_bitstream|   0|   2097152|     100|0.48566800|  PASSED  
        diehard_opso|   0|   2097152|     100|0.35772252|  PASSED  
        diehard_oqso|   0|   2097152|     100|0.47349980|  PASSED  
         diehard_dna|   0|   2097152|     100|0.48349937|  PASSED  
diehard_count_1s_str|   0|    256000|     100|0.95257057|  PASSED  
diehard_count_1s_byt|   0|    256000|     100|0.84437819|  PASSED  
 diehard_parking_lot|   0|     12000|     100|0.31899023|  PASSED  
    diehard_2dsphere|   2|      8000|     100|0.24483205|  PASSED  
    diehard_3dsphere|   3|      4000|     100|0.04900588|  PASSED  
     diehard_squeeze|   0|    100000|     100|0.96659492|  PASSED  
        diehard_sums|   0|       100|     100|0.92162735|  PASSED  
        diehard_runs|   0|    100000|     100|0.29733372|  PASSED  
        diehard_runs|   0|    100000|     100|0.21733297|  PASSED  
       diehard_craps|   0|    200000|     100|0.00620028|  PASSED  
       diehard_craps|   0|    200000|     100|0.77686961|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.41392103|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.94153849|  PASSED  
         sts_monobit|   1|    100000|     100|0.97969393|  PASSED  
            sts_runs|   2|    100000|     100|0.58348717|  PASSED  
          sts_serial|   1|    100000|     100|0.43332780|  PASSED  
          sts_serial|   2|    100000|     100|0.45215572|  PASSED  
          sts_serial|   3|    100000|     100|0.63888152|  PASSED  
          sts_serial|   3|    100000|     100|0.24532215|  PASSED  
          sts_serial|   4|    100000|     100|0.82498873|  PASSED  
          sts_serial|   4|    100000|     100|0.70631562|  PASSED  
          sts_serial|   5|    100000|     100|0.14098307|  PASSED  
          sts_serial|   5|    100000|     100|0.05971648|  PASSED  
          sts_serial|   6|    100000|     100|0.44033615|  PASSED  
          sts_serial|   6|    100000|     100|0.84254288|  PASSED  
          sts_serial|   7|    100000|     100|0.94534057|  PASSED  
          sts_serial|   7|    100000|     100|0.56895078|  PASSED  
          sts_serial|   8|    100000|     100|0.17537892|  PASSED  
          sts_serial|   8|    100000|     100|0.41596536|  PASSED  
          sts_serial|   9|    100000|     100|0.93984064|  PASSED  
          sts_serial|   9|    100000|     100|0.77199673|  PASSED  
          sts_serial|  10|    100000|     100|0.49608967|  PASSED  
          sts_serial|  10|    100000|     100|0.19713502|  PASSED  
          sts_serial|  11|    100000|     100|0.32361987|  PASSED  
          sts_serial|  11|    100000|     100|0.28265068|  PASSED  
          sts_serial|  12|    100000|     100|0.32824861|  PASSED  
          sts_serial|  12|    100000|     100|0.43554391|  PASSED  
          sts_serial|  13|    100000|     100|0.01463804|  PASSED  
          sts_serial|  13|    100000|     100|0.24260554|  PASSED  
          sts_serial|  14|    100000|     100|0.95754708|  PASSED  
          sts_serial|  14|    100000|     100|0.11090928|  PASSED  
          sts_serial|  15|    100000|     100|0.62735234|  PASSED  
          sts_serial|  15|    100000|     100|0.97145571|  PASSED  
          sts_serial|  16|    100000|     100|0.45211316|  PASSED  
          sts_serial|  16|    100000|     100|0.60723389|  PASSED  
         rgb_bitdist|   1|    100000|     100|0.36658874|  PASSED  
         rgb_bitdist|   2|    100000|     100|0.99411631|  PASSED  
         rgb_bitdist|   3|    100000|     100|0.06557851|  PASSED  
         rgb_bitdist|   4|    100000|     100|0.09920095|  PASSED  
         rgb_bitdist|   5|    100000|     100|0.69241623|  PASSED  
         rgb_bitdist|   6|    100000|     100|0.34436231|  PASSED  
         rgb_bitdist|   7|    100000|     100|0.79110397|  PASSED  
         rgb_bitdist|   8|    100000|     100|0.15949927|  PASSED  
         rgb_bitdist|   9|    100000|     100|0.82775041|  PASSED  
         rgb_bitdist|  10|    100000|     100|0.61202477|  PASSED  
         rgb_bitdist|  11|    100000|     100|0.88799469|  PASSED  
         rgb_bitdist|  12|    100000|     100|0.96488193|  PASSED  
rgb_minimum_distance|   2|     10000|    1000|0.35186805|  PASSED  
rgb_minimum_distance|   3|     10000|    1000|0.21343396|  PASSED  
rgb_minimum_distance|   4|     10000|    1000|0.18466470|  PASSED  
rgb_minimum_distance|   5|     10000|    1000|0.49211455|  PASSED  
    rgb_permutations|   2|    100000|     100|0.93013365|  PASSED  
    rgb_permutations|   3|    100000|     100|0.24389986|  PASSED  
    rgb_permutations|   4|    100000|     100|0.10744314|  PASSED  
    rgb_permutations|   5|    100000|     100|0.86806297|  PASSED  
      rgb_lagged_sum|   0|   1000000|     100|0.57089479|  PASSED  
      rgb_lagged_sum|   1|   1000000|     100|0.63991216|  PASSED  
      rgb_lagged_sum|   2|   1000000|     100|0.89554236|  PASSED  
      rgb_lagged_sum|   3|   1000000|     100|0.94219831|  PASSED  
      rgb_lagged_sum|   4|   1000000|     100|0.76276793|  PASSED  
      rgb_lagged_sum|   5|   1000000|     100|0.92606410|  PASSED  
      rgb_lagged_sum|   6|   1000000|     100|0.93189843|  PASSED  
      rgb_lagged_sum|   7|   1000000|     100|0.98320319|  PASSED  
      rgb_lagged_sum|   8|   1000000|     100|0.92024295|  PASSED  
      rgb_lagged_sum|   9|   1000000|     100|0.13888672|  PASSED  
      rgb_lagged_sum|  10|   1000000|     100|0.76576110|  PASSED  
      rgb_lagged_sum|  11|   1000000|     100|0.82383503|  PASSED  
      rgb_lagged_sum|  12|   1000000|     100|0.98684700|  PASSED  
      rgb_lagged_sum|  13|   1000000|     100|0.26130464|  PASSED  
      rgb_lagged_sum|  14|   1000000|     100|0.26742272|  PASSED  
      rgb_lagged_sum|  15|   1000000|     100|0.76937245|  PASSED  
      rgb_lagged_sum|  16|   1000000|     100|0.05400234|  PASSED  
      rgb_lagged_sum|  17|   1000000|     100|0.11081369|  PASSED  
      rgb_lagged_sum|  18|   1000000|     100|0.89407701|  PASSED  
      rgb_lagged_sum|  19|   1000000|     100|0.83792478|  PASSED  
      rgb_lagged_sum|  20|   1000000|     100|0.61427436|  PASSED  
      rgb_lagged_sum|  21|   1000000|     100|0.66549271|  PASSED  
      rgb_lagged_sum|  22|   1000000|     100|0.18433339|  PASSED  
      rgb_lagged_sum|  23|   1000000|     100|0.63460123|  PASSED  
      rgb_lagged_sum|  24|   1000000|     100|0.99377481|  PASSED  
      rgb_lagged_sum|  25|   1000000|     100|0.48356785|  PASSED  
      rgb_lagged_sum|  26|   1000000|     100|0.49883205|  PASSED  
      rgb_lagged_sum|  27|   1000000|     100|0.04619725|  PASSED  
      rgb_lagged_sum|  28|   1000000|     100|0.64940352|  PASSED  
      rgb_lagged_sum|  29|   1000000|     100|0.96496460|  PASSED  
      rgb_lagged_sum|  30|   1000000|     100|0.70586167|  PASSED  
      rgb_lagged_sum|  31|   1000000|     100|0.58423701|  PASSED  
      rgb_lagged_sum|  32|   1000000|     100|0.46024310|  PASSED  
     rgb_kstest_test|   0|     10000|    1000|0.09525747|  PASSED  
     dab_bytedistrib|   0|  51200000|       1|0.72957242|  PASSED  
             dab_dct| 256|     50000|       1|0.67061200|  PASSED  
Preparing to run test 207.  ntuple = 0
        dab_filltree|  32|  15000000|       1|0.44986941|  PASSED  
        dab_filltree|  32|  15000000|       1|0.55626134|  PASSED  
Preparing to run test 208.  ntuple = 0
       dab_filltree2|   0|   5000000|       1|0.06208234|  PASSED  
       dab_filltree2|   1|   5000000|       1|0.65494424|  PASSED  
Preparing to run test 209.  ntuple = 0
        dab_monobit2|  12|  65000000|       1|0.68823690|  PASSED  

However, bear in mind that:

Most of the tests in Diehard return a p-value, which should be uniform on [0,1) if the input file contains truly independent random bits. Those p-values are obtained by p=F(X), where F is the assumed distribution of the sample random variable X—often normal. But that assumed F is just an asymptotic approximation, for which the fit will be worst in the tails. Thus you should not be surprised with occasional p-values near 0 or 1, such as .0012 or .9983.

When a bit stream really FAILS BIG, you will get p’s of 0 or 1 to six or more places. By all means, do not, as a Statistician might, think that a p .975 means that the RNG has “failed the test at the .05 level”. Such p’s happen among the hundreds that Diehard produces, even with good RNG’s. So keep in mind that “p happens”.

Advertisements

One thought on “Running the dieharder test suite against a Fortuna implementation

  1. Colin December 27, 2014 / 11:39 pm

    I am developing a PRNG based on this same Fortuna implementation while incorporating Hot Bits. I would like you to run the same test on it if possible.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s