30

In the browser, navigating to this URL initiates a 302 (moved temporarily) request which in turn downloads a file.

http://www.targetsite.com/target.php/?event=download&task_id=123

When I view what is actually happening via Chrome network tools I see that the redirect is going to a dynamically generated path that cancels itself immediately after download. In other words, even if I know that full path I will not have time to manually call it.

So, how in using the command line can I mimic the browser actions?

I tried

curl --cookies bin/cookies.txt -O -L "http://www.targetsite.com/target.php/?event=download&task_id=123" --compressed

but this just returns gibberish. The goal of this is to programmatically download this file without having to navigate to the site. Unfortunately I cannot share the site here as it is behind a log-in.

user2029890
  • 2,493
  • 6
  • 34
  • 65

1 Answers1

72

You need to supply the -L or --location option in order to enable curl to follow HTTP redirects.

Quoting from man curl:

   -L, --location
          (HTTP/HTTPS)  If  the server reports that the requested page has
          moved to a different location (indicated with a Location: header
          and  a  3XX  response code), this option will make curl redo the
          request on the new place. If used together with -i, --include or
          -I, --head, headers from all requested pages will be shown. When
          authentication is used, curl only sends its credentials  to  the
          initial  host.  If a redirect takes curl to a different host, it
          won't be able to intercept the user+password. See  also  --loca‐
          tion-trusted  on how to change this. You can limit the amount of
          redirects to follow by using the --max-redirs option.

          When curl follows a redirect and the request is not a plain  GET
          (for example POST or PUT), it will do the following request with
          a GET if the HTTP response was 301, 302, or 303. If the response
          code  was  any  other  3xx code, curl will re-send the following
          request using the same unmodified method.
devnull
  • 118,548
  • 33
  • 236
  • 227
  • My apologies. -L is the option I tried not -R (I corrected above). It just seems to return gibberish (literally non-ascii strange characters). – user2029890 Jan 03 '14 at 14:02
  • @user2029890 Did you try omitting the `--compressed` option to see if it has any effect? – devnull Jan 03 '14 at 14:03
  • It doesn't seem to make a difference – user2029890 Jan 03 '14 at 14:30
  • I modified the original question to show the full curl command which includes cookies -L and -O. I'm not sure if I structured this correctly. Also, in your quote from the manpage it says that authentication is not followed on redirect. Can that be my problem as this is after authentication? – user2029890 Jan 03 '14 at 14:46
  • 1
    @user2029890 Did you see the `--location-trusted` option? – devnull Jan 03 '14 at 14:51
  • The --location-trusted does not seem to change anything as I'm using the cookie jar. Upon further inspection with -O it downloads a text file containing all the gibberish. It appears as if this gibberish contains info about the files being downloaded. It's a .zip file that contains .jpgs. Perhaps I just need to specify the download format differently? – user2029890 Jan 03 '14 at 17:57
  • 2
    Issue is resolved. I just had to structure the command like curl --cookie /bin/cookies.txt -L -o foo-dev.zip "http://www.targetsite.com/target.php/?event=download&task_id=123". Thanks for all your help – user2029890 Jan 03 '14 at 18:08
  • @devnull's answer is the one that actually worked. If I wanted to theorize about the "correct way to use curl" I wouldn't have come to StackOverflow. – duma May 07 '14 at 19:32
  • 1
    Just an FYI but the "gibberish" that you were seeing was most likely the contents of the file that is automatically downloaded. If you don't specify `-o ` it will just give you the response data dumped to the console. You can either redirect that data into a file with ` > output.file` or use the `-o` parameter like you are doing now. – Max Worg Aug 18 '16 at 13:54
  • In addition to `-L` modern curl has `--post301`,`--post302`, and `--post303` to use `POST` instead of `GET` after the redirect. – basin Nov 09 '18 at 09:54