2

I use a bash script

curl "$url" -o ~/Downloads/"$savePath"/"$filename" -L -C -

to download a file with support for resuming interrupted downloads (which is what -C - does). However, I don't know how to check if the download was successful and resulted in a complete file. I've tried to use --fail (-f) to get an exit status corresponding to success or failure, as follows:

curl -f "$url" -o ~/Downloads/"$savePath"/"$filename" -L -C -

However, if the download is complete, this exits with a nonzero exit status and the following error:

curl: (22) The requested URL returned error: 416

How can this be accomplished?

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
斯奇吴
  • 27
  • 8
  • http://stackoverflow.com/questions/24118224/how-to-make-asynchronous-function-calls-in-shell-scripts – sobolevn Jun 17 '16 at 15:42
  • @sobolevn, that's re: running in the background; there's no `&` here – Charles Duffy Jun 17 '16 at 15:43
  • By the way -- not all servers support request ranges, so often `-C` won't work at all; you're somewhat fortunate to be dealing with a service that supports it. – Charles Duffy Jun 17 '16 at 16:10
  • 1
    Well, I know the server supports request ranges, 'cause I built it. I write this bash script to make my TV(Well, a monitor + Mac mini) auto download something from my computer. – 斯奇吴 Jun 18 '16 at 11:17
  • @CharlesDuffy Thanks for helping me make my question better. – 斯奇吴 Jun 18 '16 at 11:28

1 Answers1

2

A HTTP 416 ("requested range not satisfiable") with -C - is a reasonable response when your file is already complete: The range of content after the current size of the file is an empty set, so while a server could return a 0-byte successful response, it can also state that no response is possible, which is what you're seeing here.

One approach you can take, if your service supports the Content-Length header, is extracting the intended file size from a HEAD request, and comparing that to the current size on-disk:

dest="$HOME/Downloads/$savePath/$filename"
if [[ -e $dest ]]; then
  remote_size=$(curl -I "$url" | awk -F: '/^Content-Length:/ { print $2 }')
  local_size=$(stat --format=%s "$dest")
  if ! [[ $remote_size ]]; then
    echo "Unable to retrieve remote size: Server does not provide Content-Length" >&2
  elif ! [[ $local_size ]]; then
    echo "Unable to check local size: Validate that GNU stat is installed" >&2
  elif (( remote_size == local_size )); then
    echo "File is complete" >&2
  elif (( remote_size > local_size )); then
    echo "Download is incomplete -- can probably resume" >&2
  elif (( remote_size < local_size )); then
    echo "Remote file shrunk -- probably should delete local and start over" >&2
  fi
else
  echo "File does not exist locally at all" >&2
fi

Note that stat --format is a GNU extension. If you're running on MacOS, you can install GNU stat as gstat via MacPorts; see BashFAQ #87 for a detailed discussion on extracting metadata if you don't have GNU tools.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Very nice answer! You nailed it down – hek2mgl Jun 17 '16 at 16:18
  • Thank you very much! I'll try that – 斯奇吴 Jun 18 '16 at 11:24
  • And, By the way, actually, You can use 'stat -f=%z "$dest"' on Mac. Thank you for asking my question. – 斯奇吴 Jun 18 '16 at 11:40
  • Hello @CharlesDuffy, I am trying your solution but I get `")syntax error: invalid arithmetic operator (error token is "` error. No changes to code – Preeti Apr 02 '23 at 14:25
  • @Preeti, that means you saved your file with DOS newlines. You need to save it as a UNIX text file -- that means a LF character at the end of each line (including the last one!), not a CRLF _separating_ lines as is done on Windows. – Charles Duffy Apr 03 '23 at 13:39
  • @CharlesDuffy thanks for the update. The file is downloaded from `curl` call. – Preeti Apr 03 '23 at 14:10
  • "The file" meaning the script you're running? It's the script itself that has the wrong newlines. Anyhow, if that _is_ the case, pipe it through `tr -d '\r'` to remove carriage returns and leave only the linefeeds, or use any of the other repair mechanisms given in the answers to [Are shell scripts sensitive to encoding and line endings?](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings) – Charles Duffy Apr 03 '23 at 16:21
  • "The file" means the actual file which will be downloaded from a server using curl call. This curl call is invoked within the script. The downloaded file is available at `$dest`. Even `ls -la $dest` shows the same size as returned by `$local_size`. – Preeti Apr 05 '23 at 10:40