1

What is the best way to read x number of binary bytes from a pipe in Perl? Using sysread returns only 8K bytes. I need to read around 1GB of bytes from the pipe before continuing with the rest of the script. Obviously I could combine all of the 8K chunks together myself, but I was hoping that something like this already exists instead of having to code it myself.

Harry Muscle
  • 2,247
  • 4
  • 38
  • 62
  • 1
    The [read](https://perldoc.pl/functions/read) function will wait if the length you specify is not yet available. Unlike sysread, it will also respect layers on the handle (such as an `:encoding` layer or `:crlf` translation on Windows), you can remove these from the handle if present with [binmode](https://perldoc.pl/functions/binmode). – Grinnz Oct 03 '19 at 16:27
  • I tried using read and it also returns 8K from a pipe unfortunately. – Harry Muscle Oct 03 '19 at 16:30
  • Perhaps it is not portable that it will do that on all OSes and pipes. I think looping until you have the amount of input you want is the best option. – Grinnz Oct 03 '19 at 16:32
  • @Grinnz, Re "*The read function will wait if the length you specify is not yet available*", It is by no means guaranteed to do that. In fact, it *doesn't* always wait when it's a pipe. [What is the difference between `read` and `sysread`?](https://stackoverflow.com/q/36315124/589924) – ikegami Oct 03 '19 at 17:40
  • 1
    How is this "off-topic"? Voting to reopen. – zdim Oct 04 '19 at 05:53

1 Answers1

3

Using sysread returns only 8K bytes

Less specifically, it returns all available bytes up to the maximum provided. (It will wait for some to arrive if there are none in the pipe.)

I need to read around 1GB of bytes from the pipe before continuing with the rest of the script.

I'm not sure there's a way to check how many bytes are waiting in the pipe, but no system has pipes that can hold 1 GB or 1 GiB. Just read in a loop.

use constant TO_READ => 1000*1000*1000;  # Or 1024*1024*1024?

my $buf = '';
while (length($buf) < TO_READ) {
   my $rv = sysread($fh, $buf, TO_READ-length($buf), length($buf));
   defined($rv)
      or die("Can't read from file: $!\n");
   $rv
      or die("Can't read from file: Premature EOF\n");
}

You could just as easily use read. What is the difference between read and sysread?

ikegami
  • 367,544
  • 15
  • 269
  • 518
  • Note that `sysread` will extend the buffer to be big enough to hold the requested number of bytes regardless of how many bytes are actually read. This means that the above program doesn't continually expand the buffer, and it end up overallocating only 8 bytes (instead of wasting up to 200 MB). – ikegami Oct 03 '19 at 17:43