7
requests.exceptions.ConnectionError: ('Connection aborted.', error(99, 'Cannot assign requested address'))

I was getting this error when running multiple processes utilizing the python requests library and calling the post function to an API that was returning very quickly (<10ms).

Dialing down the number of processes running had a delaying effect, but only dialing down to 1 process eliminated the problem. This was not a solution, but did indicate a finite resource was the culprit.

Garren S
  • 5,552
  • 3
  • 30
  • 45
  • Opening an outbound TCP connection randomly assigns a random port number in a high range (IIRC, the maximum is ~64k; where the range starts, vs ports reserved for inbound use, is configurable). Thus, if you have a really large number of outbound TCP connections open at the same time, no matter what library/programming language/etc. you're using, you're prone to hitting this issue. – Charles Duffy Jun 19 '15 at 19:59
  • 1
    This answer notes the available port range is 32768 through 61000. So number of available ports = 61000-32768+1 = 28233 http://stackoverflow.com/a/11208991/1942007 Thanks for the comment! – Garren S Jun 19 '15 at 20:18
  • Same problem here – Nathan B Feb 02 '19 at 17:37

2 Answers2

16

The way I resolved my issue was to use the requests.Session class where I would reuse the same connection/session for each call in a given process.

Contrived example:

import requests
for line in file:
  requests.get('http://example.com/api?key={key}'.format(key=line['key']))

becomes

import requests
with requests.Session() as session:
  for line in file:
    session.get('http://example.com/api?key={key}'.format(key=line['key']))

These questions had some related advice:

Repeated POST request is causing error "socket.error: (99, 'Cannot assign requested address')" Python urllib2: Cannot assign requested address

Community
  • 1
  • 1
Garren S
  • 5,552
  • 3
  • 30
  • 45
3

I too faced similar issue while executing multiple POST statements using python's request library in Spark. To make it worse, I used multiprocessing over each executor to post to a server. So thousands of connections created in seconds that took few seconds each to change the state from TIME_WAIT and release the ports for the next set of connections.

Out of all the available solutions over the internet that speak of disabling keep-alive, using with request.Session() et al., I found this answer to be working. You may need to put the header content in a separate line outside the post command though.

headers = {
        'Connection': 'close'
}
with requests.Session() as session:
response = session.post('https://xx.xxx.xxx.x/xxxxxx/x', headers=headers, files=files, verify=False)
results = response.json()
print results
Adiga
  • 121
  • 6