How to specify "socket_options" in python-requests lib since urllib3 v1.8.3 has been added the "socket_options" feature?

Question

urllib3 V1.8.3 has been added the feature "socket_options". For me, I want to specify (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) in requests lib. But I've no idea how. Here is the version change log of urllib3. Here is the sample of the new feature in urllib3:

    def test_socket_options(self):
    """Test that connections accept socket options."""
    # This test needs to be here in order to be run. socket.create_connection actually tries to
    # connect to the host provided so we need a dummyserver to be running.
    pool = HTTPConnectionPool(self.host, self.port, socket_options=[
        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    ])
    s = pool._new_conn()._new_conn()  # Get the socket
    using_keepalive = s.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE) > 0
    self.assertTrue(using_keepalive)
    s.close()

If version 1.8.3 is merged into new requests lib, how to specify the parameter. Any answers will be appreciated.

Here is my code:

import requests
import socket

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager

class SourceAddressAdapter(HTTPAdapter):
    def __init__(self, source_address, **kwargs):
        self.source_address = source_address
        super(SourceAddressAdapter, self).__init__(**kwargs)
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       socket_options=[(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)],
                                       source_address=self.source_address,
                                       )
http_session = requests.Session()
http_session.mount('http://', SourceAddressAdapter(('0.0.0.0', 1238)))
print http_session.get('http://10.0.10.7')

And here is the error:

    Traceback (most recent call last):
  File "./tt.py", line 27, in <module>
    print http_session.get('http://10.0.10.7')
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 468, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 375, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.0.10.7', port=80): Max retries exceeded with url: / (Caused by <class 'socket.error'>: [Errno 98] Address already in use)

===============================================================================

I have tried to use urllib3 v1.9 to set the (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1), but it failed. The reason why source port in time_wait state can't be reused may be "(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)" should be configured before "socket.bind". So even I set the REUSEADDR parameter in the "HTTPConnectionPool", src port in time_wait state could NOT be used. So sad..

Here is my try:

#!/usr/bin/python
import socket
from urllib3 import (
    encode_multipart_formdata,
    HTTPConnectionPool,
    )

pool = HTTPConnectionPool('10.0.10.7', 80, 
                          socket_options=[(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)],
                          source_address=('10.0.10.11',1239))
r=pool.request('GET','/')

And here is the error:

    Traceback (most recent call last):
  File "./aa.py", line 10, in <module>
    r=pool.request('GET','/')
  File "build/bdist.linux-x86_64/egg/urllib3/request.py", line 68, in request
  File "build/bdist.linux-x86_64/egg/urllib3/request.py", line 81, in request_encode_url
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 579, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 579, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 579, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 559, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/util/retry.py", line 265, in increment
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.10.7', port=80): Max retries exceeded with url: / (Caused by ProtocolError('Connection aborted.', error(98, 'Address already in use')))

Your error strongly suggests you haven't patched urllib3. That error is coming because urllib3 isn't stripping the `socket_options` kwarg, which means it isn't using it. How did you patch requests? — Lukasa, Jul 04 '14 at 10:27
After rehacking the requests, and modify my code as above. When bind the port to send request, an error occurs as above. The reason why I want to set the reuse socket option is to solve this problem(To reuse the source port in time_wait state). So I think the options doesn't work at all. I have tried the parameter socket_options in urllib3, it can work well. — bofortitude, Jul 04 '14 at 10:47
Let's be clear, you can definitely still hit "address already in use" with SO_REUSEADDR set, as you can see [here](http://stackoverflow.com/a/14388707/1401686). — Lukasa, Jul 04 '14 at 10:50
Note also that urllib3 applies the socket options _after_ creating the connection, not before. I don't think SO_REUSEADDR can work in this scenario. I've raised the question [here](https://github.com/shazow/urllib3/pull/397#issuecomment-48030826). — Lukasa, Jul 04 '14 at 10:52
What @lukasa mentions was just fixed moments ago: https://github.com/shazow/urllib3/pull/427, this will be part of v1.9 coming out early next week. — shazow, Jul 04 '14 at 22:39
@Lukasa As shazow said, if v1.9 is merged into requests, are there any changes need to be made in the code I listed above to make the "SO_REUSEADDR" work? — bofortitude, Jul 07 '14 at 02:29
@shazow I've tried (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) in urllib3 v1.9. Fail reason may be SO_REUSEADDR should be set before socket.bind. Please see the above description I added. I wonder whether this parameter could work if both source_address and socket_options are set in the same time. — bofortitude, Jul 08 '14 at 06:27
@bofortitude Ah fair point. Let's continue this conversation here and hopefully get a fix out soon: https://github.com/shazow/urllib3/pull/427 — shazow, Jul 08 '14 at 16:25

score 2 · Answer 1 · answered Jun 09 '16 at 00:03

I couldn't get @Lukasa's answer to work, but I modified it and got the following to work,

import requests
import socket
from requests.adapters import HTTPAdapter
from requests.adapters import PoolManager
from requests.packages.urllib3.connection import HTTPConnection

class SockOpsAdapter(HTTPAdapter):
  def __init__(self, options, **kwargs):
    self.options = options
    super(SockOpsAdapter, self).__init__()
  def init_poolmanager(self, connections, maxsize, block=False):
    print "init_poolmanager"
    self.poolmanager = PoolManager(num_pools=connections,
                                   maxsize=maxsize,
                                   block=block,
                                   socket_options=self.options)

options =  HTTPConnection.default_socket_options + [
              (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1),
          ]

print "build session"
s = requests.Session()
s.mount('http://', SockOpsAdapter(options))
s.mount('https://', SockOpsAdapter(options))

for i in xrange(0, 10):
  print "sending request %i" % i
  url = 'http://host:port' #put in a host/port here
  headers = {'Content-Type':'text/plain', 'Accept':'text/plain'}
  post_status = s.get(url, headers=headers)
  print "Post Status Code = %s" % str(post_status.status_code)
  print post_status.content[0:50]

Lukasa · Answer 2 · 2014-07-04T10:47:21.250

Assuming you've correctly merged urllib3 into requests and that nothing breaks, you'd use a Transport Adapter:

from requests.adapters import HTTPAdapter

class SockOpsAdapter(HTTPAdapter):
    def __init__(self, options, **kwargs):
        self.options = options

        super(SourceAddressAdapter, self).__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       socket_options=self.options)

And then you use it like this:

import requests
from requests.packages.urllib3.connection import HTTPConnection

options = HTTPConnection.default_socket_options + [
              (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1),
          ]

s = requests.Session()
s.mount('http://', SockOpsAdapter(options))
s.mount('https://', SockOpsAdapter(options))

How to specify "socket_options" in python-requests lib since urllib3 v1.8.3 has been added the "socket_options" feature?

2 Answers2