Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Great Expectations: A Critique of Current Approaches to Random Number Generation Testing & Certification

  • Conference paper
  • First Online:
Security Standardisation Research (SSR 2018)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 11322))

Included in the following conference series:

Abstract

Random number generators are a critical component of security systems. They also find use in a variety of other applications from lotteries to scientific simulations. Randomness tests, such as the NIST’s STS battery (documented in SP800-22), Marsaglia’s Diehard, and L’Ecuyer et al.’s TestU01 seek to find whether a generator exhibits any signs of non-random behaviour. However, many statistical test batteries are unable to reliably detect certain issues present in poor generators. Severe mistakes when determining whether a given generator passes the tests are common. Irregularities in sample size selection and a lack of granularity in test result interpretation contribute to this. This work provides evidence of these and other issues in several statistical test batteries. We identify problems with current practices and recommend improvements. The novel concept of suitable randomness is presented, precisely defining two bias bounds for a TRNG, instead of a simple binary pass/fail outcome. Randomness naivety is also introduced, outlining how binary pass/fail analysis cannot express the complexities of RNG output in a manner that is useful to determine whether a generator is suitable for a given range of applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The Authenticated Self project has received funding from InnovateUK under reference number 102050.

  2. 2.

    The RAMSES project has received funding from the European Union’s Horizon 2020 research and innovation program, under grant agreement No. 700326.

  3. 3.

    https://marketing.idquantique.com/acton/attachment/11868/f-004c/1/-/-/-/-/Randomness%20Test%20Report.pdf.

  4. 4.

    https://comscire.com/files/cert/comscire-pq32mu-nist_diehard-validation-tests.pdf.

  5. 5.

    https://www.commoncriteriaportal.org/files/epfiles/0712a_pdf.pdf.

  6. 6.

    https://github.com/crocs-muni/randomness-testing-toolkit.

References

  1. Schindler, W., Killmann, W.: Evaluation criteria for true (physical) random number generators used in cryptographic applications. In: Kaliski, B.S., Koç, K., Paar, C. (eds.) CHES 2002. LNCS, vol. 2523, pp. 431–449. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36400-5_31

    Chapter  MATH  Google Scholar 

  2. NXP Semiconductors Ltd. MF1PLUSx0y1 Public Datasheet. NXP Semiconductors, 21 February 2011

    Google Scholar 

  3. NXP Semiconductors Ltd. MF3D(H)x2 MIFARE DESFire EV2 contactless multi-application IC, 2 edn. NXP Semiconductors Ltd., February 2016

    Google Scholar 

  4. Altus Metrum. ChaosKey True Random Number Generator, June 2008

    Google Scholar 

  5. Marsaglia, G., Tsang, W.W., et al.: Some difficult-to-pass tests of randomness. J. Stat. Softw. 7(3), 1–9 (2002)

    Article  Google Scholar 

  6. National Institute of Standards and Technology. NIST SP800-22 Revision 1a A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications. https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-22r1a.pdf. Accessed 21 May 2018

  7. Walker, J.: Ent. A pseudo-random number sequence testing program. https://www.fourmilab.ch/random/. Accessed 07 Aug 2018

  8. L’Ecuyer, P., Simard, R.: TestU01: a C library for empirical testing of random number generators. ACM Trans. Math. Softw. (TOMS) 33(4), 22 (2007)

    Article  MathSciNet  Google Scholar 

  9. Dodis, Y., Ong, S.J., Prabhakaran, M., Sahai, A.: On the (im)possibility of cryptography with imperfect randomness. In: Proceedings of 45th Annual IEEE Symposium on Foundations of Computer Science, pp. 196–205. IEEE (2004)

    Google Scholar 

  10. Marsaglia, G.: DIEHARD, a battery of tests for random number generators. CD-ROM, Department of Statistics and Supercomputer Computations Research Institute, Florida State University. http://stat.fsu.edu/Ageo

  11. Hurley-Smith, D., Hernandez-Castro, J.: Bias in the mifare DESFire EV1 TRNG. In: Hancke, G.P., Markantonakis, K. (eds.) RFIDSec 2016. LNCS, vol. 10155, pp. 123–133. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-62024-4_9

    Chapter  Google Scholar 

  12. Hurley-Smith, D., Hernandez-Castro, J.: Certifiably biased: an in-depth analysis of a common criteria EAL4+ certified TRNG. IEEE Trans. Inf. Forensics Secur. 13(4), 1031–1041 (2018)

    Article  Google Scholar 

  13. Garcia, F.D., et al.: Dismantling MIFARE classic. In: Jajodia, S., Lopez, J. (eds.) ESORICS 2008. LNCS, vol. 5283, pp. 97–114. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88313-5_7

    Chapter  Google Scholar 

  14. Garcia, F.D., Van Rossum, P., Verdult, R., Schreur, R.W.: Wirelessly pickpocketing a Mifare Classic card. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 3–15. IEEE (2009)

    Google Scholar 

  15. Kasper, T., Silbermann, M., Paar, C.: All you can eat or breaking a real-world contactless payment system. In: Sion, R. (ed.) FC 2010. LNCS, vol. 6052, pp. 343–350. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14577-3_28

    Chapter  Google Scholar 

  16. Renesas AE45C1 and Smartcard Integrated Circuit. BSI-DSZ-CC-0212-2004 (2004)

    Google Scholar 

  17. Barker, E., Kelsey, J.: Recommendation for the entropy sources used for random bit generation (DRAFT) NIST SP800-90B (2012)

    Google Scholar 

  18. Rukhin, A., Soto, J., Nechvatal, J., Smid, M., Barker, E.: A statistical test suite for random and pseudorandom number generators for cryptographic applications. Technical report, DTIC Document (2001)

    Google Scholar 

  19. Rukhin, A., Soto, J., Nechvatal, J.: A statistical test suite for random and pseudorandom number generators for cryptographic applications. NIST DTIC Document. NIST SP800-22 (2010)

    Google Scholar 

  20. Mellado, D., Fernández-Medina, E., Piattini, M.: A common criteria based security requirements engineering process for the development of secure information systems. Comput. Stan. Interfaces 29(2), 244–253 (2007)

    Article  Google Scholar 

  21. Killmann, W., Schindler, W.: AIS 31: functionality classes and evaluation methodology for true (physical) random number generators, version 3.1. Bundesamt fur Sicherheit in der Informationstechnik (BSI), Bonn (2001)

    Google Scholar 

  22. Brown, R.G., Eddelbuettel, D., Bauer, D.: Dieharder: a random number test suite. Open Source software library, under development (2013)

    Google Scholar 

  23. Marton, K., Suciu, A.: On the interpretation of results from the NIST statistical test suite. Sci. Technol. 18(1), 18–32 (2015)

    Google Scholar 

  24. Hernandez-Castro, J., Barrero, D.F.: Evolutionary generation and degeneration of randomness to assess the indepedence of the Ent test battery. In: 2017 IEEE Congress on Evolutionary Computation (CEC), pp. 1420–1427. IEEE (2017)

    Google Scholar 

  25. Soto, J.: Statistical testing of random number generators. In: Proceedings of the 22nd National Information Systems Security Conference, vol. 10, p. 12. NIST, Gaithersburg (1999)

    Google Scholar 

  26. Turan, M.S., DoĞanaksoy, A., Boztaş, S.: On independence and sensitivity of statistical randomness tests. In: Golomb, S.W., Parker, M.G., Pott, A., Winterhof, A. (eds.) SETA 2008. LNCS, vol. 5203, pp. 18–29. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85912-3_2

    Chapter  Google Scholar 

  27. Georgescu, C., Simion, E., Nita, A.-P., Toma, A.: A view on NIST randomness tests (in)dependence. In: 2017 9th International Conference on Electronics, Computers and Artificial Intelligence (ECAI), pp. 1–4. IEEE (2017)

    Google Scholar 

  28. Zhu, S., Ma, Y., Lin, J., Zhuang, J., Jing, J.: More powerful and reliable second-level statistical randomness tests for NIST SP 800-22. In: Cheon, J.H., Takagi, T. (eds.) ASIACRYPT 2016. LNCS, vol. 10031, pp. 307–329. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-53887-6_11

    Chapter  Google Scholar 

  29. Fan, L., Chen, H., Gao, S.: A general method to evaluate the correlation of randomness tests. In: Kim, Y., Lee, H., Perrig, A. (eds.) WISA 2013. LNCS, vol. 8267, pp. 52–62. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-05149-9_4

    Chapter  Google Scholar 

  30. Hurley-Smith, D., Hernandez-Castro, J.: Quam Bene Non Quantum: Bias in a Family of Quantum Random Number Generators. Cryptology ePrint Archive, Report 2017/842 (2017). https://eprint.iacr.org/2017/842

  31. National Institute of Standards and Technology. NIST SP800-90B Reccommendation for the Entropy Sources used for Random Bit Generation. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90B.pdf. Accessed 21 May 2018

  32. Verbauwhede, I., Maes, R.: Physically unclonable functions: manufacturing variability as an unclonable device identifier. In: Proceedings of the 21st edition of the Great Lakes Symposium on VLSI, pp. 455–460. ACM (2011)

    Google Scholar 

  33. Altus Metrum. https://altusmetrum.org. Accessed 11 Sept 2018

  34. Langheinrich, M., Marti, R.: Practical minimalist cryptography for RFID privacy. IEEE Syst. J. 1(2), 115–128 (2007)

    Article  Google Scholar 

  35. Burr, W.E.: Selecting the advanced encryption standard. IEEE Secur. Priv. 99(2), 43–52 (2003)

    Article  Google Scholar 

Download references

Acknowledgements

This project has received funding from Innovate UK, under reference number 102050 (authenticatedSelf) and from the European Union’s Horizon 2020 research and innovation programme, under grant agreement No. 700326 (RAMSES project). This article is based upon work from COST Action IC1403 CRYPTACUS, supported by COST (European Cooperation in Science and Technology). We would like to thank NXP Semiconductors Ltd. for their timely and professional communication following the responsible disclosure of our findings.

Author information

Authors and Affiliations

  1. University of Kent, Canterbury, Kent, UK

    Darren Hurley-Smith & Julio Hernandez-Castro

Authors
  1. Darren Hurley-Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Julio Hernandez-Castro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Darren Hurley-Smith .

Editor information

Editors and Affiliations

  1. CISPA-Helmholtz-Zentrum (i.G.), Saarbrücken, Germany

    Cas Cremers

  2. IBM Research Zurich, Rüschlikon, Switzerland

    Anja Lehmann

Appendices

A Appendix: Test Battery Configuration

The Randomness Testing Toolkit (RTT)Footnote 6 was used to conduct the Dieharder, NIST SP 800-22 and TestU01 experiments. Ent was used as provided in the Ubuntu 16.04 LTS distribution. Modified parameters are discussed below.

1.1 A.1 Dieharder

The DESFire cards required some modification of the test parameters, due to their small sample size. The following modifications were made:

  • Overlapping Permutations: 125,000 t-samples

  • Binary rank 32x32: 4,750 t-samples

  • Binary rank 6x8: 25,000 t-samples

  • Craps: 20,000 t-samples

Four fewer tests (23 instead of 27) are conducted over 64 MB DESFire sequences. Tests that rewind during a single run are omitted. Rewinds are tolerated so long as the sequence does not reach the end of the file during a single test. 1 MB sequences are not tested with Dieharder, due to insufficient file size. All other devices use default settings over 2.1 GB.

1.2 A.2 NIST SP800-22 (revision V3)

The DESFire EV1 and DESFire EV2 cards were tested using the full NIST SP800-22 statistical test suite, but only for their 64 MB samples. The 1 MB samples were too small to allow the minimum recommended test parameters outlined in SP800-22 and so are omitted. 200 bit-streams of 1,048,576 bits were tested for each sample of every device (Quantis 16M, 4M, USB, Comscire PQ32MU, ChaosKey, urandom, DESFire EV1, and DESFire EV2). The suggested minimum for tests [6] is 100 bit-streams of at least 1,000,000 bits in length.

1.3 A.3 Ent

Ent runs for the full length of all samples. Both byte and bit tests were performed. All tests were run over all samples, without exception.

1.4 A.4 TestU01

Due to their small size, the DESFire EV1 and DESFire EV2 samples were only tested with the Alphabits and Rabbit batteries. Both batteries take an argument representing the size of the input stream, in this case the full 64MB of the samples in question.

The first \(2\cdot 10^9\) bits of the Quantis, Comscire, ChaosKey, and urandom samples were tested using Alphabits and Rabbit. Small Crush was executed using its default parameters. Crush was executed using a reduced set of tests to allow for testing 2.1 GB files (test numbers provided in brackets next to each test):

  • smarsa CollisionOver (3–10)

  • smarsa BirthdaySpacings (11)

  • snpair ClosePairs (18–20)

  • snpair ClosePairsBitMatch (21 and 22)

  • sknuth Max0ft (43)

  • svaria SampleProd (45)

  • svaria SampleMean (47)

  • svaria AppearanceSpacings (49 and 50)

  • smarsa MatrixRank (56)

  • smarsa MatrixRank (58)

  • smarsa MatrixRank (60)

  • smarsa GCD (63–64)

  • swalk RandomWalk1 (65–70)

  • scomp LinearComp (71–72)

  • scomp LempelZiv (73)

  • sspectral Fourier3 (74 and 75)

  • sstring HammingIndep (90)

  • sstring Run (91)

Big Crush was omitted, as it requires substantially more data than was included in the sample files.

B Appendix: Ent Results

See Table 5 and Figs. 2, 3.

Table 5. ENT results for the first 3 samples from each device
Full size table
Fig. 2.
figure 2

Distribution of chi-square scores for devices that fail Ent

Full size image
Fig. 3.
figure 3

Bias observed in the distribution of byte values for the first samples of devices that fail the chi-square test

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hurley-Smith, D., Hernandez-Castro, J. (2018). Great Expectations: A Critique of Current Approaches to Random Number Generation Testing & Certification. In: Cremers, C., Lehmann, A. (eds) Security Standardisation Research. SSR 2018. Lecture Notes in Computer Science(), vol 11322. Springer, Cham. https://doi.org/10.1007/978-3-030-04762-7_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-04762-7_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-04761-0

  • Online ISBN: 978-3-030-04762-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation