98

I am looking for a simple shell (+curl) check that would evaluate as true or false if an URL exists (returns 200) or not.

Jens
  • 69,818
  • 15
  • 125
  • 179
sorin
  • 161,544
  • 178
  • 535
  • 806

2 Answers2

166

Using --fail will make the exit status nonzero on a failed request. Using --head will avoid downloading the file contents, since we don't need it for this check. Using --silent will avoid status or errors from being emitted by the check itself.

if curl --output /dev/null --silent --head --fail "$url"; then
  echo "URL exists: $url"
else
  echo "URL does not exist: $url"
fi

If your server refuses HEAD requests, an alternative is to request only the first byte of the file:

if curl --output /dev/null --silent --fail -r 0-0 "$url"; then
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • 5
    Include `-o /dev/null` to suppress the printing of the header to stdout – Shawn Chin Aug 30 '12 at 14:28
  • 3
    Add --location to follow redirects and report the existence of the redirect-to URL instead of the original URL – Ivan Kozik Dec 31 '13 at 02:36
  • @IvanKozik, I'm not incorporating that only because it's up to the person implementing to decide whether it's what they really want. Often, an unexpected redirect will be indicative of a situation where the original URL isn't really valid (say, a redirect to a generic index page). – Charles Duffy Jan 05 '15 at 17:45
  • 1
    this solution does not seem to work for case 302 temporary relocation even with using -L or --location . Example: https://github.com/stedolan/jq/releases/download/jq-1.5/jq-1.5.tar.gz – Nam Nguyen Aug 20 '15 at 03:47
  • 1
    @NamNguyen, that's a server that refuses HEAD requests on principal -- ie. they don't let you see if a file is there without actually trying to download it. I'd consider that intentionally unfriendly configuration. So it's the `--head` that's a problem (against that server); the 302 has nothing to do with it. – Charles Duffy Aug 20 '15 at 16:14
  • 2
    @NamNguyen, ...it's an interesting problem to handle that gracefully -- one approach is to use `-r 0-0` to request only the first byte of the file. I've amended the answer appropriately. – Charles Duffy Aug 20 '15 at 16:22
  • @CharlesDuffy it works. Thanks for your inputs. Updated my lib with your inputs: https://github.com/gdbtek/ubuntu-cookbooks/blob/master/libraries/util.bash#L235 – Nam Nguyen Aug 20 '15 at 20:29
  • 1
    @NamNguyen As an aside -- I'd suggest avoiding the `function` keyword; it's needlessly incompatible with the POSIX sh standard, for which function declaration syntax is simply `funcname() { ... }`. – Charles Duffy Aug 20 '15 at 20:32
  • 1
    @NamNguyen Also, I'd avoid using `if ( curl ... )` in favor of `if curl ...`; the parenthesis create subshells, which makes your code less efficient. – Charles Duffy Aug 20 '15 at 20:32
  • 2
    @NamNguyen I'd also strongly suggest `return 0` and `return 1` for truthiness and falsiness, rather than using stdout to return `true` and `false` as strings. That way, someone can run: `if existURL "$url"`, with no extra (inefficient) overhead of trying to capture its stdout and interpret same. – Charles Duffy Aug 20 '15 at 20:33
  • 2
    @NamNguyen, ...if you want a grouping operator, consider `if { curl ...; }; then` insead of `if ( curl ... ); then`, which avoids this inefficiency. – Charles Duffy Aug 20 '15 at 20:35
  • @user137369, re: your suggested edit -- `${foo}` has practical benefit over `$foo` only in cases where (1) an expansion is being parameterized; or (2) a constant string is being appended to the end of an expansion's result. Please do not inject your personal preferences into others' code where the matters at hand are purely stylistic choice. – Charles Duffy Jul 09 '16 at 15:06
  • If the file being requested has zero size, nginx may give "416 Requested Range Not Satisfiable" due to "-r 0-0" – fionbio Mar 20 '17 at 17:01
  • I was trying to use the same solution with an artifactory url...and somehow it keeps on going in the else condition even for the valid URLS. – B.T Anand Feb 19 '18 at 12:49
  • How do I do this on the Windows version of curl? – Ricardo Bohner Jan 02 '20 at 21:08
  • @RicardoBohner, the `shell` tag is used for questions about POSIX-compatible shells. If you have a question about `cmd.exe` or PowerShell, ask it separately with an appropriate tag. – Charles Duffy Jan 02 '20 at 21:11
41

I find wget to be a better tool for this than CURL; there's fewer options to remember and you can actually check for its truth value in bash to see if it succeeded or not by default.

if wget --spider http://google.com 2>/dev/null; then
  echo "File exists"
else
  echo "File does not exist"
fi

The --spider option makes wget just check for the file instead of downloading it, and 2> /dev/null silences wget's stderr output.

ailnlv
  • 1,779
  • 1
  • 15
  • 29
  • The "can actually check its truth value in bash" is not unique to wget; with `--fail`, one can do the same with curl. – Charles Duffy Oct 26 '15 at 19:27
  • 3
    That's why I said "by default". Try running `wget google.com/asdf` and `curl google.com/asdf`. `curl` returns EXIT_SUCCESS by default when encountering a 404 error, while `wget` returns EXIT_FAILURE. – ailnlv Oct 28 '15 at 22:48
  • This is really slow. I check a site which has files to download in the sum of 1GB, but I don't want to download them, just check if the targets exist – rubo77 Jul 20 '18 at 09:51
  • 2
    That's what the `--spider` argument is for: it makes `wget` return after checking for the file's existence instead of downloading it. – ailnlv Jul 23 '18 at 04:55
  • I have a problem with this on github: wget --spider https://github.com/linuxwacom/libwacom/releases/download/libwacom-1.6/libwacom-1.6.tar.bz2 Leads to 403 FORBIDDEN and the message "Remote file does not exist -- broken link!!!". I needed this for a script to check whether a remote URL exists, to update programs automatically. (Interestingly, without the --spider flag, wget just downloads the file without a problem. This is weird ... ) – shevy Nov 10 '20 at 02:42
  • @shevy works fine for me using exactly the same url - I get "Remote file exists." – Pancho Aug 11 '21 at 17:46
  • A more modern approach to this would be to use --method=HEAD instead of --spider. It is *almost* the same thing internally, but is really what you want to use in this case. In a script, using `--method=HEAD` is also a more self documenting way to express the real intent – darnir Mar 09 '23 at 08:54