Parallel processing with Numpy.loadtxt()

Question

I have a >100MB file that needs to be read with numpy.loadtxt()

The reading part is the main bottleneck in my code. For a 72MB file it takes 17.3s

Is is somehow possible to read in a parallel way a file using loadtxt()

If possible without splitting the file.

score 2 · Accepted Answer · answered Mar 24 '14 at 17:52

2

It looks like numpy.loadtxt() is your problem.

According to these sites, you're better off not using numpy's load function at all.

pandas.read_csv and read_table should be helpful from the pandas module.

answered Mar 24 '14 at 17:52

horta

1

I tried `pandas.read_csv()`. It is in fact very fast in comparison with `loadtxt()`. – Tengis Mar 25 '14 at 07:20

1 Answers1