1

This is command to run my scraper.

scrapy crawl monitor -a filename="ScrapeProject.csv" -o filename.csv

It runs and saves the scraped data into filename.csv

I want to schedule as cronjob I want filename.csv to be the current datetime the scraper ran.

I tried with back-ticks but didnt work

scrapy crawl monitor -a filename="ScrapeProject.csv" -o `date`.csv

Also tried like that

scrapy crawl monitor -a filename="ScrapeProject.csv" -o "date".csv

EDIT:

Below is the command I ran upon @dps recommendation but it prompts me to enter something? 

root@ubuntu:/home/mani/pricemonitor# scrapy crawl monitor -a filename="ScrapeProject.csv" -o `date +\%m`.`date +\%d`.`date +\%y`.csv`
> 
>
Umair Ayub
  • 19,358
  • 14
  • 72
  • 146

2 Answers2

4

You aren't trying to apply any formatting to the date command in your cron job are you? You need to escape percentage signs for cron, i.e.

`date +\%m`.`date +\%d`.`date +\%y`.tar.gz

See: Percent sign % not working in crontab

Also, does it work from the command line when you don't use cron (with the ticked `date` )?

Community
  • 1
  • 1
a2f0
  • 1,615
  • 2
  • 19
  • 28
4

Scrapy Feed Exports also understand (some built-in) storage URI parameters out of the box.

%(time)s is one of them.

So you can do something like:

scrapy crawl monitor -a filename="ScrapeProject.csv" -o '%(time)s.csv'

which will create output files in the form YYYY-mm-ddTHH-MM-SS, e.g. 2017-05-11T12-12-18.csv.

Internally, time is converted using datetime.utcnow().replace(microsecond=0).isoformat().replace(':', '-').

Note: you can use any spider attribute in your Feed URI (what you set with -o). Remember that any spider argument (stuff you can add to the command line with -a key=value) will be available as spider argument (as strings).

paul trmbrth
  • 20,518
  • 4
  • 53
  • 66
  • Damnn you are so unbelievable that I had to revert my accepted answer from @dps to yours ... .... – Umair Ayub May 11 '17 at 12:46
  • 2
    **shakes fist** can't win them all, nice answer. – a2f0 May 11 '17 at 12:58
  • minor tip for windows users (tagged linux i know this was the first result!), I could only get this working with double quotes, but it works like a charm. *tips hat* – Umar.H Jun 24 '20 at 19:52