2

I have a problem, which from my perspective is some kind of special.

Iam running a system (which is not changeable) which runs the same Python script 10-100 times simultanousley. Not all the time, but when it does it, than all at once.

Actually this script, which is executes x times at the exact same moment (or just with a delay of milliseconds) needs to ask a Web API for certain data. This Web API cant handle that much requests at once (which I cant change either, nor can I modify this API in any way).

So what I would like to build, is some kind of seperate python script which runs all the time and is waiting for input from all those other scripts. This seperate script should recieve the request payload for the API, than creates a que and gets all that data. After this, is gives back the data to the python script asked for the data.

Is this somehow possible? Can someone even understand my problem? Sorry for my complicated description :D

Actually I solved this problem with an RNG in that one Script that is executed multiple times, before those scripts perform the API request, they pause for rng(x) milliseconds, so they arent execute the request all at once - but this solution is not really failproof.

Maybe there is a better solution for my problem, than my first idea.

Thanks for your help!

Dreak
  • 21
  • 1
  • Python based solution using mecurial is described by [Python: module for creating PID-based lockfile?](https://stackoverflow.com/a/1444860/2162861) Also a more OS specific approach [How to prevent a race condition when multiple processes attempt to write to and then read from a file at the same time](https://stackoverflow.com/a/30407477/2162861) – atl Sep 18 '22 at 22:40
  • Thanks for your answer! Iam not farmiliar with lockfiles, but actually I dont have a problem with acessing the same file on my system - my problem is that those scripts which will get executed all at once will perform the API request (as programmed in the script) all at once - and I would like to que it somehow. That would be an easy task if I could just run one script and doing those requests one by one, but actually Iam forced to rerun this script for every single input. If there will be a request of 100 names, there will be 100 executions at once - this is what I would like to stop. – Dreak Sep 19 '22 at 11:32
  • not sure what OS your using or how script is being invoked, I put some code together below, maybe that can demonstrate how file lock synchronization can serialize (not in any specific order) this mass invocation of scripting goodness. Best of luck! – atl Sep 22 '22 at 03:37

1 Answers1

0

fcntl.flock - how to implement a timeout?

This command executes 5 instances of a python script as fast as possible then the wait command waits for all background processes to finish.

for ((i=0;i<5;i++)) ; do ./same-lock.py &  done ; wait
[1] 66023
[2] 66024
[3] 66025
[4] 66026
[5] 66027
66025
66027
66024
66026
66023
[1]   Done                    ./same-lock.py
[2]   Done                    ./same-lock.py
[3]   Done                    ./same-lock.py
[4]-  Done                    ./same-lock.py
[5]+  Done                    ./same-lock.py

The python code below ensures that only one of those scripts runs at a time.

#!/usr/local/bin/python3

# same-lock.py

import os
from random import randint
from time import sleep
import signal, errno
from contextlib import contextmanager
import fcntl

lock_file = '/tmp/same.lock_file'

@contextmanager
def timeout(seconds):
    def timeout_handler(signum, frame):
        pass

    original_handler = signal.signal(signal.SIGALRM, timeout_handler)

    try:
        signal.alarm(seconds)
        yield
    finally:
        signal.alarm(0)
        signal.signal(signal.SIGALRM, original_handler)


# wait up to 600 seconds for a lock
with timeout(600):
    f = open(lock_file, "w")
    try:
        fcntl.flock(f.fileno(), fcntl.LOCK_EX)
        # Print the process ID of the current process
        pid = os.getpid()
        print(pid)
        # Sleep a random number of seconds (between 1 and 5)
        sleep(randint(1,5))
        fcntl.flock(f.fileno(), fcntl.LOCK_UN)
    except IOError as e:
        if e.errno != errno.EINTR:
            raise e
        print( "Lock timed out")
atl
  • 575
  • 3
  • 6