3

I'm having trouble with what I thought would be a very basic script, but has turned out to be more complicated than I imagined. I want to read data from STDIN and then write the data out to a file.

After much mucking about, I have a script which kindof works; it seems to work fine for text files (at least the MD5 sums match) but creates an unparseable file if you try it with a JPEG image.

# Start with a clean slate
rm file1

# http://unix.stackexchange.com/q/194388/5769
IFS=

#while read -r -N 8192 data; do
while read -r -N 40 data; do  # Reduced bytesize for debugging
    echo -n "$data" >> file1
done;

# Some data still remains because of how 'read' uses exit codes
echo -n "$data" >> file1

And the usage*: 

$ curl -s "http://loripsum.net/api/plaintext/5/" | ./save.sh  # Sucess
$ curl -s "http://lorempixel.com/400/200/food/"  | ./save.sh  # Failure: No error messages, but the file can't be opened with an image viewer

What's wrong with my code, and why doesn't it work for binary files?

* Yes, in this example, I could just use > to redirect the data directly to a file, but I'm eventually using this code to save POST data from an HTTP form coming in from busybox's httpd through STDIN.

IQAndreas
  • 8,060
  • 8
  • 39
  • 74
  • Just some notes on what you've done so far (don't think it will solve your problem): You can avoid `rm file` by simply redirecting the loop `done > file1`. You can also set `IFS` on the same line as `while read` to avoid changing it permanently: `IFS= while read -r -N 40 data` – Tom Fenech Apr 04 '15 at 18:01
  • Null bytes in image files cause trouble. I'm not convinced there's a way to do it, but others may have ideas that'll help you. I think `read` is the wrong tool for the job you're trying to do. You can simplify your code with `{ IFS=; while read ...; do ... done; echo ...; } > file1` with a single I/O redirection. Or you can use `exec >file1` so standard output all goes to the file. – Jonathan Leffler Apr 04 '15 at 18:03
  • Why not just rely on `curl` to fetch and save the output? – anubhava Apr 04 '15 at 18:06
  • @anubhava In this particular example that would work, but I'll eventually use this code to save POST data from an HTTP form through STDIN coming in from busybox's httpd. – IQAndreas Apr 04 '15 at 18:07
  • Be aware that binary data can sometimes get corrupted when you pass it through text utilities. – Peter Bowers Apr 04 '15 at 18:08
  • @PeterBowers What would be a "text utility" in this particular case, `read` or `echo`? And is there a way to avoid them? (_EDIT:_ Ah, I see the answer you posted avoids them both!) – IQAndreas Apr 04 '15 at 18:10
  • `cat > file1` in the script would read from `STDIN` and write data to "file1" until EOF. Is there a particular reason you are trying to read in chunks? – Michael - sqlbot Apr 04 '15 at 18:11
  • @Michael-sqlbot Because I couldn't find any flag like `read --all` (and I believe writing code like `read -N 9999999` [makes American Indians cry](https://www.youtube.com/watch?v=j7OHG7tHrNM)). – IQAndreas Apr 04 '15 at 18:12
  • d'oh the answer below precedes me. I didn't see it. I think `cat` is prolly all you need. – Michael - sqlbot Apr 04 '15 at 18:15

2 Answers2

4

If you want to accept from STDIN and output to a file, this works pretty well...

#!/usr/bin/bash
cat >file1
Peter Bowers
  • 3,063
  • 1
  • 10
  • 18
1

echo does not deal with binary data properly. See this answer for details. You may be better advised to use a scripting language like perl if you want to do anything more than simple redirection.

Community
  • 1
  • 1
Neil Masson
  • 2,609
  • 1
  • 15
  • 23