python-How to make sure the script crawls periodically?

Question

I have written a python script that crawls a website and creates a json response file with the crawled data. Now, I want to use the json response but how do I make sure that the script executes periodically so that the json file is not static but keeps changing as the website which is being crawled changes it data?

I'd look at using `cron` if you have it or if you're on Windows without cygwin you could look at `at` or apparently that's been deprecated in favor of `schtasks` — Eric Renouf, Nov 28 '15 at 15:23

score -1 · Answer 1 · answered Nov 28 '15 at 15:23

-1

I believe you are looking for CRON.

The software utility Cron is a time-based job scheduler in Unix-like computer operating systems.

https://en.wikipedia.org/wiki/Cron

answered Nov 28 '15 at 15:23

Loïc

11,804
1
31
49

score -1 · Answer 2 · edited May 23 '17 at 12:15

-1

import schedule
import time

def job():
    print("I'm working...")

schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)

while 1:
    schedule.run_pending()
    time.sleep(1)

How do I get a Cron like scheduler in Python?

edited May 23 '17 at 12:15

Community

1
1

answered Nov 28 '15 at 15:27

Ricky Wilson

3,187
4
24
29

python-How to make sure the script crawls periodically?

2 Answers2