0

I would like to read a fixed number of bytes from stdin of a python script and output it to one temporary file batch by batch for further processing. Therefore, when the first N number of bytes are passed to the temp file, I want it to execute the subsequent scripts and then read the next N bytes from stdin. I am not sure what to iterate over in the top loop before While true. This is an example of what I tried.

import sys
While True:
    data = sys.stdin.read(2330049) # Number of bytes I would like to read in one iteration
    if data == "":
        break
    file1=open('temp.fil','wb') #temp file
    file1.write(data)
    file1.close()
    further_processing on temp.fil (I think this can only be done after file1 is closed)
Vishnu
  • 499
  • 1
  • 5
  • 23

1 Answers1

0

Two quick suggestions:

  1. You should pretty much never do While True
  2. Python3

Are you trying to read from a file? or from actual standard in? (Like the output of a script piped to this?)

Here is an answer I think will work for you, if you are reading from a file, that I pieced together from some other answers listed at the bottom:

with open("in-file", "rb") as in_file, open("out-file", "wb") as out_file:
    data = in_file.read(2330049)
    while byte != "":
        out_file.write(data)

If you want to read from actual standard in, I would read all of it in, then split it up by bytes. The only way this won't work is if you are trying to deal with constant streaming data...which I would most definitely not use standard in for.

The .encode('UTF-8') and .decode('hex') methods might be of use to you also.

Sources: https://stackoverflow.com/a/1035360/957648 & Python, how to read bytes from file and save it?

AtHeartEngineer
  • 363
  • 4
  • 15
  • Thank you for the response but I'm trying to read from an actual standard in. I saw the responses you mentioned earlier but I can't seem to figure out how i could do it with the regular standard in. Perhaps, I'm missing something obvious – Vishnu Dec 01 '17 at 07:47
  • Can you read all the data in at once? Then subdivide the data into bytes? – AtHeartEngineer Dec 01 '17 at 07:56
  • Nope, I cannot do that. I have a continuous stream of data like you indicated in your initial answer. What do you think is the best practice if I have constant streaming data? – Vishnu Dec 01 '17 at 08:04
  • 1
    Honestly, a ring/circular buffer, in C or C++. I'm sure this is doable in python somehow, but for anything dealing with actual bytes, standard in, or raw buffers...I'd go with C or C++. Maybe this will help? https://codereview.stackexchange.com/questions/85068/efficient-ring-buffer-fifo – AtHeartEngineer Dec 01 '17 at 08:15