Generate large random text files with python and NumPy

Question

For testing data, I am in need of quickly creating large files of random text. I have one solution, taken from here and given below:

import random
import string

n = 1024 ** 2  # 1 Mb of text
chars = ''.join([random.choice(string.letters) for i in range(n)])

with open('textfile.txt', 'w+') as f:
    f.write(chars)

My problem is that this takes 653 ms to perform, way too much for my uses.

Is there a faster way to quickly generate text files with random text?

Possible duplicate of [Generating random text strings of a given pattern](https://stackoverflow.com/questions/367586/generating-random-text-strings-of-a-given-pattern) — frozen, Jul 15 '17 at 20:54
You can put random.choice(string.letters) for i in range(n) into generator and use yield to make it faster — dmitryro, Jul 15 '17 at 20:54
The duplicate suggests exactly my example, and as mentioned it is too slow. The use case would be to generate random text for automated testing (the main goal is to do regression testing of performance on a parser). — Jonas Adler, Jul 15 '17 at 20:57
@JonasAdler I timed an approach with numpy that gets this down to 370ms. Is that still too slow? — cs95, Jul 15 '17 at 21:01
@dmitryro I cannot pass a generator to `f.write`, and using a generator inside `''.join()` gives no performance increase. But I do agree it improves the code. — Jonas Adler, Jul 15 '17 at 21:02
If 370ms with numpy is the best we got, I'll have to make do. But I was hoping for an order of magnitude. — Jonas Adler, Jul 15 '17 at 21:03
You're asking Python to do something fast, which is not what Python does. Pick a different language, like C. — Alex Reynolds, Jul 15 '17 at 21:04
But python does lots of things fast, I don't see why I should be stuck with simple loops. At the very least this should be vectorizable? — Jonas Adler, Jul 15 '17 at 21:07

cs95 · Accepted Answer · 2017-07-15T21:18:14.073

1

Create a numpy array of letters:

In [662]: letters = np.array(list(chr(ord('a') + i) for i in range(26))); letters
Out[662]: 
array(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
       'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'],
      dtype='<U1')

Use np.random.choice to generate random indices b/w 0 and 26, and index letters to generate random text:

np.random.choice(letters, n)

Timings:

In [664]: n = 1024 ** 2

In [701]: %timeit np.random.choice(letters, n)
100 loops, best of 3: 15.1 ms per loop

Alternatively,

In [705]: %timeit np.random.choice(np.fromstring(letters, dtype='<U1'), n)
100 loops, best of 3: 14.1 ms per loop

edited Jul 15 '17 at 21:18

answered Jul 15 '17 at 21:06

cs95

379,657
97
704
746

I am able to modify this somewhat and get an order of magnitude better performance: `np.random.choice(np.fromstring(string.letters, dtype='S1'), n)`, total time `17 ms`. Could you update the answer to that and I'll accept that answer? – Jonas Adler Jul 15 '17 at 21:10
1

@JonasAdler That gives you a list of chars, right? You'll want to join them together. – cs95 Jul 15 '17 at 21:13
It seems `f.write` accepts char arrays. The result looks alright and writing is basically instant. – Jonas Adler Jul 15 '17 at 21:13
@JonasAdler I got you a bit faster, if you don't mind the fact they're not binary strings. – cs95 Jul 15 '17 at 21:16
@JonasAdler Glad to help :) – cs95 Jul 15 '17 at 21:18

Generate large random text files with python and NumPy

1 Answers1

Timings:

Linked