-6

I have a URL from which I have to download data on daily basis, e.g.: www.manishshukla.com/files/05-17-2016.csv

In the above URL, it gives me data for 17th May 2016. Similarly for the data of 18th May 2016, URL will be: www.manishshukla.com/files/05-18-2016.csv

I want Bash script, which will automatically takes the date of the day and downloads the file.

Please help me how to do it. I want to add this job on cron also, so that I don't need to manually run the code.

Ani Menon
  • 27,209
  • 16
  • 105
  • 126
Manish
  • 33
  • 1
  • 8
  • 6
    Honestly, who has upvoted this awful question. There is no specific problem, they are just asking someone to write a script for them – 123 May 17 '16 at 09:27

2 Answers2

2

You can use this:

wget www.manishshukla.com/files/"$(date '+%d-%m-%Y')".csv

If you want to save it somewhere specific, then

wget www.manishshukla.com/files/"$(date '+%d-%m-%Y')".csv -P /path/to/dir

You can put this command in a daily cron to download the files in a daily basis. For cron, use the full path of wget i.e /usr/bin/wget or /usr/local/bin/wget or whatever it is.

Jahid
  • 21,542
  • 10
  • 90
  • 108
  • Can you please provide me the complete code, from start as well as saving of downloaded file. – Manish May 17 '16 at 09:09
2

You have two bits here. One is to schedule a cron job and the other is to get the file & save it.

Steps :

  1. Open the terminal & run crontab -e

    minute(0-59) hour(0-23) day(1-31) month(1-12) weekday(0-6) command

  2. In place of minute, hour, day, month & weekday. Also provide the command to run.

  3. The command to put up here is : wget www.manishshukla.com/files/"$(date '+\%d-\%m-\%Y')".csv

  4. Save the file & the jobs are scheduled.

In your case :

0 0 * * * wget --quiet -O www.manishshukla.com/files/"$(date '+\%d-\%m-\%Y')".csv

OR

0 0 * * * /usr/bin/curl www.manishshukla.com/files/"$(date '+\%d-\%m-\%Y')".csv

This would run the command every day at 0hrs 0mins.

Added \ to escape %, as it may not work in corntab without escaping.

Percent-signs (%) in the command, unless escaped with backslash (), will be changed into newline characters, and all data after the first % will be sent to the command as standard input.

Ani Menon
  • 27,209
  • 16
  • 105
  • 126
  • I understood scheduling it on cron, I need the complete bash script. So that I can run it in my terminal also for previous dates also. – Manish May 17 '16 at 09:21
  • It starts from today & goes on. If you want it for the previous dates then just write a for loop to put appropriate date and get the files with `wget` or `curl` – Ani Menon May 17 '16 at 10:06
  • Note this will probably not work: you need to escape the `%` in the cron expression: [Why percent signs (%) do not work in crontab?](http://stackoverflow.com/q/16238460/1983854). – fedorqui May 17 '16 at 10:55
  • 1
    @fedorqui Thanks, added it to the answer. – Ani Menon May 17 '16 at 14:07