Compress files while reading data from STDIN

Question

Is it possible to compress (create a compressed archive) data while reading from stdin on Linux?

I'm voting to close this question as off-topic because it belongs to Unix & Linux — Dan Dascalescu, Jun 30 '19 at 10:53

score 218 · Accepted Answer · edited Feb 21 '17 at 15:43

218

Yes, use gzip for this. The best way is to read data as input and redirect the compressed to output file i.e.

cat test.csv | gzip > test.csv.gz

cat test.csv will send the data as stdout and using pipe-sign gzip will read that data as stdin. Make sure to redirect the gzip output to some file as compressed data will not be written to the terminal.

edited Feb 21 '17 at 15:43

Alex Riley

169,130
45
262
238

answered Nov 18 '09 at 19:27

Space

7,049
6
49
68

5

Note that the test.csv.gz files remains of 0 size until the data flux is closed. – MUY Belgium Apr 26 '13 at 10:00
... and how does one do that? Is it really a separate question? – russellpierce Mar 08 '15 at 14:26
13

This also works with `xz`: `mysqldump mydb | xz > dbdump.sql.xz` – jeroen Jul 27 '15 at 18:32
3

Could use `-9`, default compression level is `-6`. – Pablo Bianchi Oct 05 '17 at 13:48
1

@MUY Belgium: not exactly, it starts writing only after 16KB of compressed blocks. See [here](https://unix.stackexchange.com/questions/643078/tool-library-to-compress-decompress-on-the-fly/643084#643084) – Francois Oct 05 '21 at 15:37

score 113 · Answer 2 · answered Nov 18 '09 at 19:00

113

Yes, gzip will let you do this. If you simply run gzip > foo.gz, it will compress STDIN to the file foo.gz. You can also pipe data into it, like some_command | gzip > foo.gz.

answered Nov 18 '09 at 19:00

jtbandes

115,675
35
233
266

score 1 · Answer 3 · answered Nov 18 '09 at 18:58

1

gzip > stdin.gz perhaps? Otherwise, you need to flesh out your question.

answered Nov 18 '09 at 18:58

retracile

12,167
4
35
42

Compress files while reading data from STDIN

3 Answers3