0

I'm trying to get information from a webservice which provides ~300 endpoints every 5 seconds, but each endpoint call take ~0.5sec

The issue is I'm falling behind the webservice since it take more time to handle every call than generating new ones.

I created threaded function with threading module and it works, but there is a limitation on simultaneous calls on the webservices.

Is there a way to limit threading to for example 10?

def print_blocks(blocks):
    for block in blocks: <-- each block is generated every 5 sec
        for tsx in block["transactions"]: <-- each block contains ~300 transactions/webservices calls
            try:
                # print_tsx is a function with the webservice call and print the needed values
                threading.Thread(target=print_tsx, args=(tsx["hash"],)).start()
mkrieger1
  • 19,194
  • 5
  • 54
  • 65
ExploiterZ
  • 17
  • 8
  • 1
    You're looking for [Semaphore](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore) – JL0PD Jun 24 '21 at 10:18
  • 1
    Maybe `asyncio` is a better/easier way to do the processing? https://docs.python.org/3/library/asyncio.html and https://docs.python.org/3/library/asyncio-task.html – Jan Christoph Terasa Jun 24 '21 at 10:19
  • 1
    Or if you prefer threads, create exactly 10 threads and make them read from a [`Queue`](https://docs.python.org/3/library/queue.html#queue.Queue). – bereal Jun 24 '21 at 10:20

0 Answers0