1

I trying to poll an API and then time how long it takes to make a file available

def main():
    pool = ThreadPoolExecutor(max_workers=8)
    for filename in os.listdir(data_dir):
      pool.submit(poll_status, filename)

def poll_status(filename: str):
    start_time = perf_counter()
    for i in range(30):
        logging.info(f"Attempt {i}")
        resp = requests.get(f"https://api/{filename}/statuses")
        if resp.status_code == 404:
            # Wait 3 seconds and try again on next attempt
            time.sleep(3)
        elif resp.status_code == 200:
            end_time = perf_counter()
            logging.info(
                f'{filename} took {end_time - start_time: 0.2f} seconds to complete. (Attempt {i})')

Printing times taken for each file to become available seem to be increasing while the attempts are not.

INFO:root:f1 took  0.19 seconds to complete. (Attempt 0)
INFO:root:f2 took  4.79 seconds to complete. (Attempt 0)
INFO:root:f3 took  7.43 seconds to complete. (Attempt 0)
INFO:root:f4 took  7.99 seconds to complete. (Attempt 0)
INFO:root:f5 took  12.11 seconds to complete. (Attempt 0)
INFO:root:f6 took  15.84 seconds to complete. (Attempt 0)
INFO:root:f7 took 16.54 seconds to complete. (Attempt 0)
INFO:root:f8 took  16.35 seconds to complete. (Attempt 0)

If an attempt took 16 seconds I would expect it to be Attempt 5? (as there should be a 3 second wait after each attempt)

Am I using Thread Pool incorrectly or is there a problem with how I am timing each attempt?

Anon957
  • 539
  • 2
  • 6
  • 24
  • 2
    Your calculation of the wait time from the code excluded any latency that `request.get` may take to return a value, which includes all the time that it takes for a response be returned by the server. See [thread](https://stackoverflow.com/questions/6579127/delay-a-task-until-certain-time) describing the general problem and [thread](https://stackoverflow.com/questions/23267409/how-to-implement-retry-mechanism-into-python-requests-library) for a solution specific to `requests`. – metatoaster Jan 06 '23 at 02:41

0 Answers0