2

I've got a file located at an url, say https://www.example.com/file.csv

I'm required to download the file every time it's updated (or at least once/24h) and perform certain actions on the file via scripts.

However, I don't have any kind of notification service / webhook to inform me whenever the file is reuploaded on the server (I've even spoken to support and I can't get what I need).

How can I poll the link for changes? So that every time the file updates, my scripts fire up the download and proceed with the tasks?

I thought curl can help me, but I'm totally lost on it and some google searches have not given me the results I need.

remus2232
  • 77
  • 8
  • Possible duplicate of [PHP - Email notification whenever a remote file changes](https://stackoverflow.com/questions/9495512/php-email-notification-whenever-a-remote-file-changes) – Satish Saini Feb 22 '19 at 20:58

1 Answers1

6

curl can help!

The easiest way to do this with curl is to simply invoke it automatically like every 12 hours or so (preferably then using crontab or similar). Then ask curl to only get the file if the remote file is newer than the local one:

curl -O -z file.csv https://www.example.com/file.csv

And if you'd do it in a crontab, you would possibly add a line similar to:

0 0,12 * * * cd download-dir && curl -O -z file.csv https://www.example.com/file.csv

... which makes curl update the file accordingly in the dedicated download directory, every 12th hour - if the file is newer on the server than on disk.

How does it work?

As the documentation for curl -z explains:

The date expression (passed to -z) can be all sorts of date strings or if it doesn't match any internal ones, it is taken as a filename and tries to get the modification date (mtime) from the file instead.

Daniel Stenberg
  • 54,736
  • 17
  • 146
  • 222
  • How does this get the file only if it’s newer? The -z flag seems like it should have a date expression after it but I don’t see a date in your answer. – styfle Feb 24 '19 at 16:51