2

I'm working on parallel GET functionality with apply_async as described in Python Requests: Don't wait for request to finish. My problem is that I need to supply headers to every GET request and I fail to figure out how to do that.

I'm trying along these lines:

items.append(pool.apply_async(requests.get, [url, "", {"header1":"value1", "header2":"value2"}]))

and many variations of the theme without success.

I would appreciate information how to work my way through this.

Thanks!

Squrppi
  • 95
  • 1
  • 7

1 Answers1

2

According to the requests docs, you need to pass headers in the headers keyword argument to requests.get.

According to the multiprocessing docs, the arguments to apply_async are:

apply_async(func, args=(), kwds={}, callback=None)

Which in your case would translate to:

pool.apply_async(requests.get, 
                 [url], 
                 dict(headers={"header1":"value1", "header2":"value2"}))
larsks
  • 277,717
  • 41
  • 399
  • 399
  • Thanks! I tried {headers:{... but it did not work. This dict(headers={... works. – Squrppi Sep 04 '17 at 12:56
  • Well, `dict(headers={...})` produces the same result as `{'headers': {...}}`, so I'm not sure that was the problem, but I'm glad things are working! – larsks Sep 04 '17 at 13:40