Curl is uncompressing a compressed file when I didn't ask it to

Question

This is the opposite of the issue that all my searches kept coming up with answers to, where people wanted plain text, but got compressed.

I'm writing a bash script that uses curl to fetch the mailing list archive files from a Mailman mailing list (using the standard Mailman web interface on the server end).

The file (for this month) is http://lists.example.com/private.cgi/listname-domain.com/2013-September.txt.gz (sanitized URL).

When I save this with my browser I get, in fact, a gzipped text file, which when ungzipped contains what I expect.

When I fetch it with Curl (after previously sending the login password and getting a cookie set, and saving that cookie file to use in the request), though, what comes out stdout (or is saved to a -o file) is the UNCOMPRESSED text.

How can I get Curl to just save the data into a file like my browser does? (Note that I am not using the --compressed flag in my Curl call; this isn't a question of the server compressing data for transmission, it's a question of downloading a file that's compressed on the server disk and I want to keep it compressed.)

(Obviously I can hack around this by re-compressing it in my bash script. Waste of CPU resources, and a problem waiting to happen in the future, though. Or I can leave it uncompressed, and hack the name and store it as just September.txt; that wastes disk space instead. Again, that would break if the behavior changed in the future, though. The problem seems to me to be that Curl is getting confused between compressed transmittal, and and actual compressed data.)

How are you verifying that the file has been ungzipped? (I know that sounds like a weird question, but if the answer is "I looked at it", then with what tool?) (Or, to be less mysterious, if you looked at the file with `less`, try `less -L`) — rici, Oct 01 '13 at 05:12
possible duplicate of [How to properly handle a gzipped page when using curl?](http://stackoverflow.com/questions/8364640/how-to-properly-handle-a-gzipped-page-when-using-curl) — Jayan, Oct 01 '13 at 08:50
What is your command line? curl by default does not alter file. "curl -o /tmp/afile ", did get compressed file. (The file is exact binary replica ) — Jayan, Oct 01 '13 at 09:03
please give the output of executing `curl -vs >/dev/null` this will help us see what is in the response that tells curl to decompress the file. — SingleNegationElimination, Oct 01 '13 at 11:46

score 5 · Answer 1 · answered Oct 01 '13 at 11:40

5

Is it possible the server is decompressing the file based on headers sent (or not sent) by curl? Try the following header with curl:

--header 'Accept-Encoding: gzip,deflate'

answered Oct 01 '13 at 11:40

Cole Tierney

9,571
1
27
35

score 3 · Answer 2 · edited May 23 '17 at 11:58

3

You can download the *.txt.gz directly, without any uncompressing, with 'wget' instead of 'curl'.

wget http://lists.example.com/private.cgi/listname-domain.com/2013-September.txt.gz

If curl is essential, then check out the details here

edited May 23 '17 at 11:58

Community

1
1

answered Oct 01 '13 at 08:15

philshem

24,761
8
61
127

Curl is uncompressing a compressed file when I didn't ask it to

2 Answers2