120

I have a web api http://something.com/api and I want to use GET to get the response body.

This is my command:

curl "http://something.com/api"

Of course, it fails and gives an error message.

When I use Chrome and input the above url, everythings correct. However I do the same things with Firefox, the url gives me the same error message. I try to repeat the action with Chrome extension DHC, the request gives correct response again. After some searching, I believe that the curl option --user-agent makes a difference. What is the correct way to set the user agent to Chrome? Or this is not the point, the problem comes from other fields? Thank you very much.

Hermann Schwarz
  • 1,495
  • 1
  • 15
  • 30
wdetac
  • 2,712
  • 3
  • 20
  • 42

3 Answers3

166

If you need to set the user header string in the curl request, you can use the -H option to set user agent like:

curl -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36" http://stackoverflow.com/questions/28760694/how-to-use-curl-to-get-a-get-request-exactly-same-as-using-chrome

Updated user-agent form newest Chrome at 02-22-2021

Using a proxy tool like Charles Proxy really helps make short work of something like what you are asking. Here is what I do, using this SO page as an example (as of July 2015 using Charles version 3.10):

  1. Get Charles Proxy running
  2. Make web request using browser
  3. Find desired request in Charles Proxy
  4. Right click on request in Charles Proxy
  5. Select 'Copy cURL Request'

Copy cURL Request example in Charles 3.10.2

You now have a cURL request you can run in a terminal that will mirror the request your browser made. Here is what my request to this page looked like (with the cookie header removed):

curl -H "Host: stackoverflow.com" -H "Cache-Control: max-age=0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36" -H "HTTPS: 1" -H "DNT: 1" -H "Referer: https://www.google.com/" -H "Accept-Language: en-US,en;q=0.8,en-GB;q=0.6,es;q=0.4" -H "If-Modified-Since: Thu, 23 Jul 2015 20:31:28 GMT" --compressed http://stackoverflow.com/questions/28760694/how-to-use-curl-to-get-a-get-request-exactly-same-as-using-chrome
Feriman
  • 503
  • 8
  • 17
Mike Grace
  • 16,636
  • 8
  • 59
  • 79
42

Open Chrome Developer Tools, go to Network tab, make your request (you may need to check "Preserve Log" if the page refreshes). Find the request on the left, right-click, "Copy as cURL".

hoodslide
  • 631
  • 6
  • 5
  • 2
    But cookie in "Copy as cURL" expires within few minutes. At least in case of most of the sites. How i can automate it to fetch new cookie ? – Gaurav Kansal Apr 23 '20 at 03:29
  • Similar option is available in FireFox: open Developer Tools -> Network tab, then right-click on the request and open "Copy Value" sub-menu – Payam Sep 15 '22 at 17:47
9

Check the HTTP headers that chrome is sending with the request (Using browser extension or proxy) then try sending the same headers with CURL - Possibly one at a time till you figure out which header(s) makes the request work.

curl -A [user-agent] -H [headers] "http://something.com/api"

MegaAppBear
  • 1,220
  • 1
  • 9
  • 11
  • I am not familiar with it. Can you give me some real examples for setting user-agent to chrome and possible headers? – wdetac Mar 03 '15 at 07:05
  • 1
    For anyone who may not familiar with this. `curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" "http://something.com/api"`. – vee Apr 26 '22 at 04:22