1

Is it possible to pipe numpy data (from one python script ) into the other?

suppose that script1.py looks like this:

x = np.zeros(3, dtype={'names':['col1', 'col2'], 'formats':['i4','f4']})

print x

Suppose that from the linux command, I run the following:

python script1.py | script2.py

Will script2.py get the piped numpy data as an input (stdin)? will the data still be in the same format of numpy? (so that I can, for example, perform numpy operations on it from within script2.py)?

user3262424
  • 7,223
  • 16
  • 54
  • 84

3 Answers3

3

Check out the save and load functions. I don't think they would object to being passed a pipe instead of a file.

Wang
  • 3,247
  • 1
  • 21
  • 33
2

No, data is passed through a pipe as text. You'll need to serialize the data in script1.py before writing, and deserialize it in script2.py after reading.

Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • A quick-and-dirty way to do the serialization is to use the python `repr` function. Replace `print x` with `print repr(x)` in script1.py and then have script2.py call `eval` to de-serialize the output of script1.py – srgerg Mar 11 '11 at 02:56
  • Thank you guys. I have a numpy array that needs to be written to a binary file, and I just thought piping would make the whole process faster (i.e, one process computes, one process writes). Do you have any other ideas to make this work faster? – user3262424 Mar 11 '11 at 03:30
  • `mmap` makes the world go 'round. – Ignacio Vazquez-Abrams Mar 11 '11 at 03:36
0

See this question.

If you're willing to use the subprocess module, you can share memory between processes to pass numpy arrays rapidly. If not, I've found saving to a file beats the pants off of piping, probably because converting the array to a string is so slow.

Community
  • 1
  • 1
Andrew
  • 2,842
  • 5
  • 31
  • 49