Curl error when downloading json object

Question

getting the following error...

curl: (56) GnuTLS recv error (-54): Error in the pull function.

...when using the following command to curl a json file

curl -L -o commerce.json http://www.commerce.gov/data.json

Any advice? I'm not familiar with curl. Perhaps it's a timeout error. Is there anyway I can prevent that? I really need the file, and I am unable to download it from a browser (assuming too big a file).

I'm working from command line on Ubuntu. Would love, alternatively to curl, a python solution.

And why not read the documentation for curl that describes this? — Мона_Сах, Sep 01 '16 at 20:02

score 2 · Answer 1 · edited May 23 '17 at 12:34

2

the error code 56 means the following, as described here https://curl.haxx.se/docs/manpage.html

56 Failure in receiving network data.

you should use a -v to see what's happen.

I don't thnik that another tool fix the network error.

Never the less there is a example in plain python.

CURL alternative in Python

edited May 23 '17 at 12:34

Community

1
1

answered Sep 01 '16 at 19:24

Aleksandar

2,442
3
15
24

ode2k · Answer 2 · 2016-09-02T15:13:53.807

1

In bash you can use:

wget -O commerce.json http://www.commerce.gov/data.json

Otherwise, a Python solution to this would be:

First you will need to install the Python wget library, then you can use the code:

import wget
url = 'http://www.commerce.gov/data.json'

commercejson = wget.download(url)

This will download the data.json file to your local Python project directory. The data.json file is currently 198MB, so curl may not be able to handle it very well.

UPDATE: Compressed JSON download:

To enable gzip compression, you can use the following to download gzipped version, which ends up being 19MB instead, which would be much more friendly to download.

wget -S --header="accept-encoding: gzip" -O commerce.json.gz http://www.commerce.gov/data.json

Then, once the gzipped json file is downloaded, run the below command to decompress it:

gzip -d commerce.json.gz

edited Sep 02 '16 at 15:13

answered Sep 01 '16 at 19:48

ode2k

2,653
13
20

Does wget work for you? I get endless tries like so: --2016-09-01 18:41:59-- https://www.commerce.gov/sites/commerce.gov/files/data.json Reusing existing connection to [www.commerce.gov]:443. HTTP request sent, awaiting response... 200 OK Length: 206604792 (197M) [application/json] Saving to: ‘commerce2.json’ commerce2.json 0%[ ] 127.67K 749KB/s in 0.2s 2016-09-01 18:42:59 (749 KB/s) - Read error at byte 130732/206604792 (Connection reset by peer). Retrying. – The_Last_Question Sep 01 '16 at 22:51
wget does work for me, but I have a very fast internet connection. – ode2k Sep 02 '16 at 15:09
I updated my answer to enable gzip compression on the downloaded json data. – ode2k Sep 02 '16 at 15:14

Curl error when downloading json object

2 Answers2