2

I am having an issue with my curl request. I am getting (through a GET request) a JSON from a web server and it works really great in my web browser. However when I get it from curl (or my ruby code), some value of my JSON are empty (they were not in my browser). That makes me lose a lot of data which is not acceptable. Does anyone have an idea of what makes this happen ? I already tried to change the user agent (see below) and still it gives the same result.

curl "http://..../path/to/json"

or

curl -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36" "http://..../path/to/json"

gives the same result.

EDIT : I tried copying curl request from Safari (see below) and it gives the same missing values. Chrome however display the same result as curl.

curl 'https://path/to/json' -XGET -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4'
Aeradriel
  • 1,224
  • 14
  • 36
  • Can you include a sample response in the question? – Ravi Chandra Jul 26 '17 at 10:03
  • `-H` places a Header. If you want to change the user-agent, you have to use the `-A` flag. `curl -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" "http://..../path/to/json"` – bam Jul 26 '17 at 10:03
  • 1
    you should take a look at the accepted answer here https://stackoverflow.com/questions/28760694/how-to-use-curl-to-get-a-get-request-exactly-same-as-using-chrome , other header fields may be required other than User Agent alone. It really depends on the server you are requesting. – minhhn2910 Jul 26 '17 at 10:03
  • Even copying curl request from my browser (Safari), I have the same missing values. Chrome request gives however the same result as curl... – Aeradriel Jul 26 '17 at 10:13
  • your 2nd curl command is malformed and shouldn't work. Cache-Control is its own header, yet is missing -H – hanshenrik Jul 26 '17 at 10:18
  • Yeah I failed my copy/paste here but there was a -H when I typed it in my terminal. – Aeradriel Jul 26 '17 at 10:20

2 Answers2

1

i guess its possible that your browser ignore the content-length header, and just keeps reading until the connection is closed. curl isn't like that by default (probably for performance reasons, because it can be much slower to read until the connection is dropped), and stops reading once content-length bytes has been read. and that your target webserver is bugged, and sends the wrong length. you can tell curl to ignore the header and read everything, with the --ignore-content-length switch.

another possibility is that the webpage you're reading the curl from, fills in the blanks with javascript somehow?

another possibility is that the full json is not displayed from the target server without having a http session cookie first?

hanshenrik
  • 19,904
  • 4
  • 43
  • 89
0

It seems that the problem came from the fact that the response is customize depending on the language of the browser. So in French the json is full and in English some fields are missing...

Aeradriel
  • 1,224
  • 14
  • 36