python execute function every n seconds wait for completion

Question

I want to execute FUNCTION periodically every 60 seconds, but I don't want to execute FUNCTION again IF the previous run has not completed yet. If the previous run completes in e.g. 120s then I want to execute a new FUNCTION call straight away. If previous run completed in e.g. 10s then I want to wait 50s before I execute a new FUNCTION call.

Please see my implementation below.

Can I achieve it with e.g. subprocess.run or some timeloop library so that the implementation would be much cleaner?

import time


def hello(x):
    # some logic here
    # execution could take any time between e.g. <10s, 120s>


def main(ii):
    while True:
        start = int(time.time())

        try:
            val = next(ii)
        except StopIteration as ex:
            return None

        else:
            hello(val)

            run_time_diff = int(time.time()) - start

            if run_time_diff < 60:
                time.sleep(60 - run_time_diff)


ii = iter(list[[...],[...],...[...]])
main(ii=ii)

There is no need for the low-level `iter` and `next`. Just use a plain `for` loop, it is much more cleaner. I think your scheduler with runtime calculation is OK, except the `int()` for the timestamps introduces rounding errors. — VPfB, Mar 16 '20 at 08:45
I just want to operate on the iterator due to memory efficiency. — Dariusz Krynicki, Mar 16 '20 at 08:48
Did you check [run-certain-code-every-n-seconds](https://stackoverflow.com/questions/3393612/run-certain-code-every-n-seconds) ? — Albin Paul, Mar 16 '20 at 08:50
I saw it and I could not see a solution there to handle the drift. — Dariusz Krynicki, Mar 16 '20 at 08:53
@NullByte A for-loop uses an iterator internally, no need to duplicate its functionality. Passing a list as an argument does not decrease memory efficency, the list (or any other object) will NOT be copied. — VPfB, Mar 16 '20 at 09:07
is it not the case that I hold a variable with list as a value in a memory and I keep the whole list in memory while I iterate over it? If I create iterator out of it then the memory usage drops? hmm although I assume I still hold reference to the original list...so you may be right. Am I assuming correctly? — Dariusz Krynicki, Mar 16 '20 at 09:13
@NullByte You can save memory with iterators only if you generate the list item by item or so. If the list is given, the iterator is just a moving index existing in addition to the original list. — VPfB, Mar 16 '20 at 17:12

score 0 · Answer 1 · answered Mar 17 '20 at 13:37

maybe apsheduler could help you. But if your job wil run more then waiting time, it could be skipped. In this case you can increase number of workers.

import datetime
import time

from apscheduler.schedulers.background import BackgroundScheduler

scheduler = BackgroundScheduler()

def some_job():
    time.sleep(5)
    print(f"{datetime.datetime.utcnow()}: Every 10 seconds")


job = scheduler.add_job(some_job, 'interval', seconds=10, max_instances=1)

scheduler.start()
try:
    while True:
        time.sleep(1)
finally:
    scheduler.shutdown()

python execute function every n seconds wait for completion

1 Answers1