How to schedule a script to run periodically?

Question

I have a task to write a Python script which has to parse a web-page once a week. I wrote the script but do not know how can I make it to work once a week. Could someone share an advice and write possible solution?

Answer 1

Have a look at cron. Its not python, but fits the job much better in my opinion. For example:

@weekly python path/to/your/script

A similar question was discussed here.

Answer 2

Whether the script itself should repeat a task periodically usually depends on how frequently the task should repeat. Once a week is usually better left to a scheduling tool like cron or at.

However, a simple method inside the script is to wrap your main logic in a loop that sleeps until the next desired starting time, then let the script run continuously. (Note that a script cannot reliably restart itself, or showing how to do so is beyond the scope of this question. Prefer an external solution.)

Instead of

def main():
    ...


if __name__ == '__main__':
    main()

use

import tim


one_week = 7 * 24 * 3600  # Seconds in a week

def main():
    ...


if __name__ == '__main__':
    while True:
        start = time.time()
        main()
        stop = time.time()
        elapsed = stop - start
        time.sleep(one_week - elapsed)

Answer 3

Are you planning to run it locally? Are you working with a virtual environment?

Task scheduler option

If you are running it locally, you can use Task scheduler from Windows. Setting up the task can be a bit tricky I found, so here is an overview:

1. Open Task Scheduler > Create Task (on right actions menu)
2. In tab "General" name the task
3. In tab "Triggers" define your triggers (i.e. when you want to schedule the tasks)
4. In tab "Actions" press on new > Start a program. Under Program/script point to the location (full path) of your python executable (python.exe). If you are working with a virtual environment it is typically in venv\Scripts\python.exe. The full path would be C:\your_workspace_folder\venv\Scripts\python.exe. Otherwise it will be most likely in your Program Files.
5. Within the same tab, under Add arguments, enter the full path to your python script. For instance: "C:\your_workspace_folder\main.py" (note that you need the ").
6. Press Ok and save your task.

Debugging

To test if your schedule works you could right click on the task in Task scheduler and press on Run. However, then you don't see the logs of what is happening. I recommend therefore to open a terminal (eg cmd) and type the following:

C:\your_workspace_folder\venv\Scripts\python.exe "C:\your_workspace_folder\main.py"

This allows you to see the full trace of your code and if its running properly. Typical errors that occur are related to file paths (eg if you are not using the full path but a relative path).

Sleeping mode

It can happen that some of the tasks do not run because you don't have administrator privileges and your computer goes in sleeping mode. What I found as a workaround is to keep the computer from going into sleeping mode using a .vbs script. Simply open notepad and create a new file named idle.vbs (extension should be .vbs so make sure you select all programs). In there paste the following code:

Dim objResult

Set objShell = WScript.CreateObject("WScript.Shell")    

Do While True
  objResult = objShell.sendkeys("{NUMLOCK}{NUMLOCK}")
  Wscript.Sleep (60000)
Loop