simple async with callback request python 3

Question

I'm new in python I'm try to develop app that send about 10,000 request to test my API, i use pyqt5 to and python 3, i want to make many A-sync request using requests , send the many requests and not wait for response but add callback when response is ready for example update UI, please give me a working example,thanks for help

i already use multi-threads but is seems sync and slow can you give any approach — Fath Bakri, Dec 23 '19 at 14:17
what did you try? [aiohttp](https://aiohttp.readthedocs.io/en/stable/), [aiohttp-requests](https://github.com/maxzheng/aiohttp-requests). You can also try to use `threading` or `multiproccessing` There could be even modules `requests-threading` or `requests-multiproccessing` — furas, Dec 23 '19 at 14:23
BTW: I never used it with `Qthread` but yesterday was question for [aiohttp](https://stackoverflow.com/questions/59439708/running-url-requests-in-parallel-with-flask/59442785#59442785) — furas, Dec 23 '19 at 14:27
Working with `Qthread` it may need to use queue to send messages to main thread because usually GUI has to run in main thread. And main thread may uses some `QTimer` to periodically get values from queue and update GUI - without stoping `mainloop` (and freezing GUI) — furas, Dec 23 '19 at 14:30
can you give me a example of send asyn request vi requests and add callback when response is ready, this is first step then i will try to use it in Qthread, also i read multi-thread is not good approach for send fast request but a syn/wait is recommended https://stackoverflow.com/questions/2632520/what-is-the-fastest-way-to-send-100-000-http-requests-in-python — Fath Bakri, Dec 23 '19 at 14:39
can any one give me an example of send asyn request vi requests and add callback when response is ready — Fath Bakri, Dec 23 '19 at 14:53
`requests` can't send async request - you have use `requests` +`thread` or `multiprocess` to send many requests at the same time. But if you use `aiohttp` then you can create async request but then you don't need module `requests`. And in [link to question for aiohttp](https://stackoverflow.com/questions/59439708/running-url-requests-in-parallel-with-flask/59442785#59442785) you would have to add callback to `fetch()` to run it when it get response. — furas, Dec 23 '19 at 16:33

score 0 · Accepted Answer · answered Dec 23 '19 at 18:25

Hi furas try your code and it worked without add proxies to session.post here is the code

import aiohttp
import asyncio
import sys
import time
from decimal import localcontext, Decimal, ROUND_HALF_UP
from datetime import datetime,timezone ,timedelta
BASE_URI = 'https://d.com/session.php'

def show_exception_and_exit(exc_type, exc_value, tb):
        traceback.print_exception(exc_type, exc_value, tb)
        sys.exit(-1)


async def fetch(session, url):
    proxies = {
        "http"  :"http://W09s1H:eMwmFL@91.188.240.67:9951",
        "https" :"https://W09s1H:eMwmFL@91.188.240.67:9951"
        }
    Timestamp = int((datetime.utcnow() - datetime(1970,1,1)).total_seconds())
    headers = {
        "user-agent": "Mozilla/5.0 (X11; Linux i686; rv:64.0) Gecko/20100101 Firefox/64.0"
        }
    JsonSession = "{'device':'id'}"

    async with session.post(url,
                            data=JsonSession,
                            headers=headers,
                            params={"timestamp":Timestamp}) as response:
        return await response.text()

async def main(url):
    tasks = []
    results = []
    async with aiohttp.ClientSession() as session:
        for n  in range(1,10):
            tasks.append(fetch(session, url))
        results = await asyncio.gather(*tasks)
    return results

def parallel(url):
    loop = asyncio.get_event_loop()
    results = loop.run_until_complete(main(url))
    return results

# --- main ---



result = parallel(BASE_URI)

for item in result:
    print(item[:300])
    print('-----')

and when i add proxies like this :

 async with session.post(url,
                            data=JsonSession,
                            headers=headers,
                            params={"timestamp":Timestamp},
                            proxies=proxies) as response:

i get error

Traceback (most recent call last):
  File "C:/Users/soft/AppData/Local/Programs/Python/Python37-32/MyApp/Scripts/Test.py", line 51, in <module>
    result = parallel(BASE_URI)
  File "C:/Users/soft/AppData/Local/Programs/Python/Python37-32/MyApp/Scripts/Test.py", line 44, in parallel
    results = loop.run_until_complete(main(url))
  File "C:\Users\soft\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 579, in run_until_complete
    return future.result()
  File "C:/Users/soft/AppData/Local/Programs/Python/Python37-32/MyApp/Scripts/Test.py", line 39, in main
    results = await asyncio.gather(*tasks)
  File "C:/Users/soft/AppData/Local/Programs/Python/Python37-32/MyApp/Scripts/Test.py", line 30, in fetch
    proxies=proxies) as response:
  File "C:\Users\soft\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\client.py", line 876, in post
    **kwargs))
TypeError: _request() got an unexpected keyword argument 'proxies'

how to add proxies to session.post because i really need to add it

i add proxy like this : async with session.post(url, data=JsonSession, headers=headers, params={"timestamp":Timestamp}, proxy=httpproxy) as response: , and it worked fine thanks for help — Fath Bakri, Dec 23 '19 at 19:09

simple async with callback request python 3

1 Answers1