Python 3 threading post request passing header params and data

Question

I'm trying to make my post requests faster because at the moment it takes 3 seconds per post. As I need to iterate it n times it could take hours. So, I started looking for threading, async calls and many others but none solved my problem. Mostly of the problems was due to the fact that I couldn't specify the headers and the params of my post request.

My Python version is 3.6.7

My code:

for i in range(0, 1000):
  assetId = jsonAssets[i]['id']
  uuidValue = uuid.uuid4()

  headers = {'Content-Type': 'application/json',}
  params = (('remember_token', '123456'),)

  data = ('{{"asset":'
          '{{"template":1,'
          '"uuid":"{uuidValue}", '
          '"assetid":{assetId}}}}}'
          .format(uuidValue = uuidValue, 
                  assetId = assetId))
  response = requests.post('http://localhost:3000/api/v1/assets', headers=headers, params=params, data=data)

Some of the tries were using:

pool.apply_async

or

ThreadResponse

But I couldn't set headers or params like in the request.post

So, how can I make this post request using this header, params and data faster?

Thanks in advance and sorry for any trouble, this is my first stackoverflow post.

score 1 · Accepted Answer · answered Jan 06 '19 at 16:07

1

If you can make single request properly, shortest way for you is to use ThreadPoolExecutor:

def single_request(i):
  assetId = jsonAssets[i]['id']
  uuidValue = uuid.uuid4()
  # ... all other requests stuff here

  return response


with ThreadPoolExecutor(max_workers=10) as executor:
    futures = {
        executor.submit(single_request, i): i
        for i 
        in range(1000)
    }

    for future in as_completed(futures):
        i = futures[future]
        try:
            res = future.result()
        except Exception as exc:
            print(f'excepiton in {i}: {exc}')
        else:
            print(res.text)

answered Jan 06 '19 at 16:07

Mikhail Gerasimov

36,989
16
116
159

Hello Mikhail, I tried your solution and after I run it using range 100 the total time spent to execute this task just decreased 11 seconds from my previous code, going from 121 seconds to 110 seconds. – Rafael Lobo Jan 07 '19 at 13:28
@RafaelLobo well, if you're requesting localhost, you won't see any significant change. When you start to request some real domain, you should see real results. [Here's detailed explanation why](https://stackoverflow.com/a/51180506/1113207). In case however you don't see increase on real domain, I would need reproducible code to help. – Mikhail Gerasimov Jan 07 '19 at 13:40
Thanks Mikhail, I think that link explained the problem. I will try my requests in another server. – Rafael Lobo Jan 07 '19 at 14:21
1

Mikhail, I just tested it in an server that could handle more than one request at a time and it was 70% faster than my localhost server, which can only handle 1 request at a time. Thanks for helping! – Rafael Lobo Jan 07 '19 at 20:24

score 0 · Answer 2 · answered Jan 04 '19 at 22:14

0

You should use async libraries like aiohttp

answered Jan 04 '19 at 22:14

ipave

1,249
9
17

Python 3 threading post request passing header params and data

2 Answers2