8

I asked a question about how to throttle a python upload, which sent me to this answer, where I was informed of a little helper library called socket-throttle. That's all fine and dandy for regular HTTP and probably also for most plain uses of the socket. However, I'm trying to throttle an SSL connection, and trying to combine socket-throttle with the stock SSL library (used implicitly by requests) causes an exception deep in the guts of the library:

  File "***.py", line 590, in request
    r = self.session.get(url, headers=extra_headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 394, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 324, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 478, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 95, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util.py", line 643, in ssl_wrap_socket
    ssl_version=ssl_version)
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 211, in __init__
    socket.__init__(self, _sock=sock._sock)
  File "***/socket_throttle.py", line 54, in __getattr__
    return getattr(self._wrappedsock, attr)
AttributeError: '_socket.socket' object has no attribute '_sock'

Well, that's a downer. As you can tell, the ssl package is trying to use one of the socket's private fields, _sock rather than the socket itself. (Isn't the point of private fields that you're not supposed to access them from the outside? Grr.) If I try to inject myself into that field on my ThrottledSocket object, I run into this problem:

    File "/home/alex/dev/jottalib/src/jottalib/JFS.py", line 590, in request
    r = self.session.get(url, headers=extra_headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 394, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 324, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 478, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 95, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util.py", line 643, in ssl_wrap_socket
    ssl_version=ssl_version)
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 241, in __init__
    ciphers)
TypeError: must be _socket.socket, not ThrottledSocket

Now what? Is there somewhere else in this where I could rate-limit the python communication? Or is there a cleaner way to do it than having to override the socket implementation? Which turns out to be moot anyway, since the ssl package just tries to bypass it altogether.

Community
  • 1
  • 1
Alex
  • 3,429
  • 4
  • 36
  • 66
  • The reason `ssl` is rooting around in private `socket` fields is, the underlying C library for TLS, `openssl`, wants very much to talk directly to the OS-level socket descriptor. What might work instead is changing `socket-throttle` so that it monkeypatches `ssl.wrap_socket` *instead of* `socket.socket` -- you need the throttle wrapper *outside* the TLS wrapper. I'm not going to post that as an answer because I don't know whether it'll work, and even if it does it'll probably be a lot of tinkering. Good luck? – zwol Feb 01 '15 at 21:56
  • The ssl library is not mucking around in the private guts of a `socket.socket` object. `ssl.wrap_socket` returns a new object which stores the original instance of `socket.socket` on it as `_sock`. It has every right to be using its own private attribute. Next time, please read the source before making statements like that about other libraries. Also, as a point of order, the exception is coming from `ssl`/`socket-throttle` but bubbles up through requests. Requests has no responsibility for this. – Ian Stapleton Cordasco Feb 02 '15 at 14:31

2 Answers2

4

Depending on your requirements, you can and maybe should solve this particular problem on the OS level instead of on the application level.

Approaching this on the OS level has two advantages. First, it does not make a difference how the sockets involved are used (HTTP or HTTPS or IRC or some ping of death packets -- it does not matter). Secondly, the more you decouple the different components of your system, the easier it is to make changes afterwards and to debug issues.

There are tools (at least for POSIX-compliant systems) for throttling bandwidth of network interfaces and/or processes. You might want to have a look at these, for example:

  • trickle (for shaping traffic of processes)
  • wondershaper (for shaping traffic of entire network interfaces, I have actually used this from within a modern Ubuntu, and it works perfectly fine)

These discussions might be relevant for you:

Community
  • 1
  • 1
Dr. Jan-Philip Gehrcke
  • 33,287
  • 14
  • 85
  • 130
  • I was hoping to provide the user with a configurable upload (and possibly download) setting in the program itself. The trickle solution looks like it would solve my problem, but it also removes control of the situation from my program to an external entity... still, I'm really starting to lose hope about this situation. Either Python simply *can't do* what I'm asking, or it's just so complicated or esoteric that nobody on Stack Overflow knows how. Honestly I thought this would be a relatively common scenario, but I'm starting to think I was wrong in that assumption. – Alex Feb 05 '15 at 18:00
  • Taking control over network traffic surely *is* possible, but it is much less trivial than most people think. The TCP/IP stack is incredibly complex and well-engineered, incorporating many special algorithms. Your operating system does a very good job of hiding this complexity from you. You can achieve what you want, just not in a short project. Maybe this make you some home: If trickle works for you, you can ship it together with your application, and put everything into a "wrapper", whereas the wrapper configures trickle as well as your app. – Dr. Jan-Philip Gehrcke Feb 05 '15 at 18:25
  • By the way, you may want to upvote those answers that provided you with some insight, even if it wasn't the perfect solution that you hoped to obtain. ;) – Dr. Jan-Philip Gehrcke Feb 05 '15 at 18:27
  • True. I sometimes forget. – Alex Feb 05 '15 at 18:47
  • Oh, and you might want to check out https://github.com/praus/shapy which I just came across. – Dr. Jan-Philip Gehrcke Feb 06 '15 at 16:39
1

It looks like you're trying to throttle HTTP requests. If that's the case, you can try RequestsThrottler instead. Python requests is way nicer than httplib too.

kichik
  • 33,220
  • 7
  • 94
  • 114
  • I'm actually using the requests library (which in turn uses httplib). This `RequestsThrottler` seems promising, but I'll have to test it to make sure before I accept your answer. :) I wasn't having issues with throttling HTTP connections, but HTTPS connections is where I've been having trouble. – Alex Feb 02 '15 at 14:25
  • Either I don't understand how RequestsThrottler works, or it only affects downloads, not uploads. I need to throttle uploads, as the first line in my question says. Any chance you could provide me with a working sample, if you've gotten it to work before? – Alex Feb 05 '15 at 07:42