Are there any ready made libraries so that the intel hardware prng (rdrand) can be used by numpy programs to fill buffers of random numbers?
Failing this can someone point me in the right direction for some C code that I could adapt or use (I use CPython and Cython with numpy so the bare minimum wrapper shd be enough).
The random generators I want are uniform random numbers between [0,1).
This code will use /dev/urandom (Unix) or CryptGenRandom APIs (Windows). Which RNG is used, hardware or software, is dependent on the operating system.
If you want to control exactly which generator is used, you must query it through its hardware driver or library. When you have the random bits as a string, you proceeed similarly to this code, using np.fromstring.
Normally we can trust the operating system to use the best entropy sources for its cryptographic services, including the random bit generator. If there is a hardware RNG it is likely to be used, usually combination with other entropy sources.
import os
import numpy as np
def cryptorand(n):
a = np.fromstring(os.urandom(n*4), dtype=np.uint32) >> 5
b = np.fromstring(os.urandom(n*4), dtype=np.uint32) >> 6
return (a * 67108864.0 + b) / 9007199254740992.0
Here is the distribution of 1,000,000 random deviates generated with this method on Mac OS X. As you can see it is quite uniform on [0,1):
If you need really strong cryptographic random deviates, you can use /dev/random instead of /dev/urandom. This applies only to Unix-like systems, not Windows:
import numpy as np
def cryptorand(n):
with open('/dev/random','rb') as rnd:
a = np.fromstring(rnd.read(n*4), dtype=np.uint32) >> 5
b = np.fromstring(rnd.read(n*4), dtype=np.uint32) >> 6
return (a * 67108864.0 + b) / 9007199254740992.0
Note that this function might block, unlike the version that uses os.urandom as entropy source.
(Edit1: Updated normalization to equate that of NumPy)
Edit 2: The comments indicates the reason for the question was speed, not cryptographic strength. The purpose of hardware RNG is not speed but strength, so it makes the question invalid. However, a fast and good PRNG which can be an alternative to the Mersenne Twister is George Marsaglia's multiply-with-carry generator. Here is a simple implementation in Cython:
import numpy as np
cimport numpy as cnp
cdef cnp.uint32_t gw = 152315241 # any number except 0 or 0x464fffff
cdef cnp.uint32_t gz = 273283728 # any number except 0 or 0x9068ffff
def rand(cnp.intp_t n):
"""Generate n random numbers using George Marsaglia's
multiply-with-carry method."""
global gw, gz
cdef cnp.uint32_t w, z, a, b
cdef cnp.intp_t i
cdef cnp.ndarray x = cnp.PyArray_SimpleNew(1, &n, cnp.NPY_FLOAT64)
cdef cnp.float64_t *r = <cnp.float64_t*> cnp.PyArray_DATA(x)
w,z = gw,gz
for i in range(n):
z = 36969 * (z & 65535) + (z >> 16)
w = 18000 * (w & 65535) + (w >> 16)
a = (z << 16) + w
z = 36969 * (z & 65535) + (z >> 16)
w = 18000 * (w & 65535) + (w >> 16)
b = (z << 16) + w
r[i] = (a * 67108864.0 + b) / 9007199254740992.0
gw,gz = w,z
return x
Beware that neither Mersenne Twister nor multiply-with-carry have cryptographic strength.
There is a paper available here (http://iopscience.iop.org/article/10.3847/1538-4357/aa7ede/meta;jsessionid=A9DA9DDB925E6522D058F3CEEC7D0B21.ip-10-40-2-120) and (non-paywalled version) here (https://arxiv.org/abs/1707.02212) that describes how to use Intel Secure Key, which is a cryptographically secure random number generator implemented on-chip. It can be accessed via RdRand and RdSeed instructions.
But the author seems to say you should go for implementing it in C/C++ instead of Python. The rdrand python module runs about 100x slower than the Python default random number generator, and about 1000x slower than the one in Numpy (paper section 5.2).
