Python threading Timer not filling output file

Question

I am using the module threading class Timer. I want to set up a process that will be repeated every day, generating an output json file every time is launched (day).

The problem is that it generates a file that is NOT filled until the whole process finishes (so if it has to run for an entire year, I shall wait all the year).

Here the code.

from processing import *
import sys
from datetime import datetime, timedelta
from threading import Timer

if __name__ == "__main__":
    '''
    '''
    #Function that generates a json file, that (works), but only is filled if Timer is NOT runing
    geojson_gen(sys.argv[1],
                sys.argv[2],
                sys.argv[3],
                out_filename = 'test_country'
                )

    for rep in range(10000):
        #Get the number of seconds for the next time Timer will launch the func. geojson_gen(). Here some code to get the number of seconds in which it shall be launched the timer
        next_date = (launch_date - start_date).seconds

        t = Timer(next_date , geojson_gen(sys.argv[1],
                                    sys.argv[2],
                                    sys.argv[3],
                                    out_filename = 'test_country'
                                    )
                 )
        t.start()

So all the script runs properly and if I comment all the Timer part, I get the json file. But when I start the process to run everyday, it generates empty json files (not filled).

What is wrong? How to get the json filled after the function geojson_gen() finished (and not after the whole Timer process)?

Thanks a lot!

Could you expand on your question? What is the value of `diff`? Is `next_date` used/needed? Do you want a start a short process each day that generates a file and is then done? Or do want to append to a file every day? Or do you want to start a new process every day, that can run for many days, creates 1 file per process that is appended daily? Does `geojson_gen` use time sensitive functions? — J.D., May 25 '20 at 17:32
Oh thanks for the answer: - You are true, diff = next_date. It is just that I am showing the most critical part of the code - The process must start a new process every day, generating a json file (.geojson) for each day. - geojson_gen() use NO sensitive functions, just sorting and classifying data (numpy - loops) and once it finished it close the .geojson file (file_out.close()) - The point is that it generates a file everyday, but the data is retained in RAM, so you can't visualize it in the geojson. If I use it without Timer() it works perfectly — joueswant, May 26 '20 at 18:03

score 1 · Accepted Answer · answered May 27 '20 at 06:09

With the code given I don't know why empty files are generated, but starting lots of Timers in parallel is not a good solution for your goal. Because what you want is not a parallel process, but a repeated, sequential one.

A more efficient way is to create a time-based loop. This example shows the basic idea:

import time

#set the wait time (a day in seconds)
waitTime = 60*60*24 

# set initial value
startTime = time.time()

# runs forever
while True:
    # check how much time has passed
    timeDiff = time.time() - startTime
    # if a day has passed, generate a json and update starttime
    if timeDiff > waitTime:
        geojson_gen()
        startTime = time.time()

This means you have to leave the script running though. An even cleaner solution is to only use the script that creates the json, then schedule in your OS to run that script daily. See this answer for a Windows example.

score 0 · Answer 2 · answered Aug 18 '22 at 08:34

I had a similar issue with a script that uses threading.Timer(), the script was simply doing print(). This would work from the terminal, but when piping the output to a file, that file would only get updated very rarely. This then meant I couldn't do tail -f. This was solved by adding sys.stdout.flush(). Not sure if this helps the OP, but it may be a hint in the right direction.

Python threading Timer not filling output file

2 Answers2