Curl command returning file regardless if it exists or not

Question

I am running a curl command that passes cookies.txt with an authentication string generated in a previous step and attempts to download a file that is generated daily. This works great when the file exists, but the problem i'm running into is when the file is not yet released. I start listening at 5PM and I re-run the script every 5 minutes and attempt to grab the file. Currently I check to see if the file size is above a certain value, but it doesn't work very well.

Is there any way to set curl to only create a file if the file it's attempting to grab exists? I'd really like to avoid the whole checking file size practice since that's really unreliable for multiple files of differing sizes.

Curl Command:

curl -b cookies.txt -J -L -v -O https://file_http_example.thespot.com/cleared_product_$(date +"%Y_%m_%d").xlsx

I try comparing the file size by doing the following:

fileSize=500
targetFileSize=$(wc -c file_name_$(date + "%Y_%m_%d").xlsx | awk '{print $1}')
if [ "$targetFileSize" -gt "$fileSize" ]
then keep the file otherwise delete it.

UPDATE: I just ran the script with the below changes and I still saved a file. This is what's saved in the file:

@Aaron I think you mean `-f`. `-F` is for sending form data. — Barmar, Nov 04 '20 at 20:14
And it looks like that works. Many servers send an error page along with 404 errors, and without that option you download the error page. — Barmar, Nov 04 '20 at 20:15
@Barmar that's an option too, done. I'll delete my comments then. Thanks for testing it :) — Aaron, Nov 04 '20 at 20:19

score 1 · Answer 1 · answered Nov 04 '20 at 20:16

1

Assuming the server respond with an HTTP error when the file isn't available, you can use curl's -f flag :

-f, --fail          Fail silently (no output at all) on HTTP errors

This will avoid creating a file when the server responds with for an HTTP 404 or other error code.

answered Nov 04 '20 at 20:16

Aaron

24,009
2
33
57

Testing this during execution this evening. If it works will accept as answer. Thank you! @Aaron – StormsEdge Nov 04 '20 at 20:22
I just ran it manually and unfortunately it did not work. Please take a look at the edited above. – StormsEdge Nov 04 '20 at 20:37
Can you rerun the command without the -f and with -v ? I'd like to know which HTTP code the server responds with – Aaron Nov 04 '20 at 20:56
But given the file's content it looks like this is an HTTP 200, which explains why `-f` doesn't work. I'll see if I can find something else – Aaron Nov 04 '20 at 20:57
After a bit of search I don't think you have a better choice than your manual filesize check, the problem is that from `curl`'s point of view everything went alright – Aaron Nov 04 '20 at 21:11
A bit far-fetched, but you could try adding an `Accept: application/octet-stream` (or maybe [`application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`](https://stackoverflow.com/a/4212908/1678362) ?) header to your request. I doubt it'll work, but it could make the server raise an error when all it has to serve is HTML rather than your xlsx file – Aaron Nov 04 '20 at 21:17

Curl command returning file regardless if it exists or not

1 Answers1