Bandwidth throttling using Twisted

Question

I'm trying to set speed limits on downloading/uploading files and found that twisted provides twisted.protocols.policies.ThrottlingFactory to handle this job, but I can't get it right. I set readLimit and writeLimit, but file is still downloading on a maximum speed. What am I doing wrong?

from twisted.protocols.basic import FileSender
from twisted.protocols.policies import ThrottlingFactory
from twisted.web import server, resource
from twisted.internet import reactor
import os

class DownloadPage(resource.Resource):
    isLeaf = True

    def __init__(self, producer):
        self.producer = producer

    def render(self, request):
        size = os.stat(somefile).st_size
        request.setHeader('Content-Type', 'application/octet-stream')
        request.setHeader('Content-Length', size)
        request.setHeader('Content-Disposition', 'attachment; filename="' + somefile + '"')
        request.setHeader('Accept-Ranges', 'bytes')

        fp = open(somefile, 'rb')
        d = self.producer.beginFileTransfer(fp, request)

        def err(error):
            print "error %s", error

        def cbFinished(ignored):
            fp.close()
            request.finish()
        d.addErrback(err).addCallback(cbFinished)

        return server.NOT_DONE_YET


producer = FileSender()
root_resource = resource.Resource()
root_resource.putChild('download', DownloadPage(producer))
site = server.Site(root_resource)
tsite = ThrottlingFactory(site, readLimit=10000, writeLimit=10000)
tsite.protocol.producer = producer
reactor.listenTCP(8080, tsite)
reactor.run()

UPDATE

So sometime after I run it:

2012-10-25 09:17:03+0600 [-] Unhandled Error
Traceback (most recent call last):
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 402, in startReactor
        self.config, oldstdout, oldstderr, self.profiler, reactor)
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 323, in runReactorWithLogging
        reactor.run()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
        self.mainLoop()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
        self.runUntilCurrent()
    --- <exception caught here> ---
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 334, in unthrottleWrites
        p.unthrottleWrites()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 225, in unthrottleWrites
        self.producer.resumeProducing()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 919, in resumeProducing
        self.consumer.unregisterProducer()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/web/http.py", line 811, in unregisterProducer
        self.transport.unregisterProducer()
      File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 209, in unregisterProducer
        del self.producer
    exceptions.AttributeError: ThrottlingProtocol instance has no attribute 'producer'

I see that I'm not supposed to assign producer like I do know tsite.protocol.producer = producer, I'm new to Twisted and I don't know how to do that another way.

Looking at the source there is a line, `log.msg("Throttling reads on %s" % self)` can you verify that this is being logged? — John, Oct 24 '12 at 12:03
it doesn't log *throttleReads methods, but it does *throttleWrites: `Throttling writes on ` — dan.ch, Oct 25 '12 at 03:19
I reckon you have to insert Throttling object either between `file` read and and `render()`, or perhaps better between `reactor` and your `render/DownloadPage instance`. Right now you seem to pass `producer` to both DownloadPage and Throttling objects, this doesn't seem right. — Dima Tisnek, Oct 27 '12 at 17:32
`ThrottlingFactory` takes factory as an argument and since the only factory instance here is `site` I passed it. If there is a working example of bandwidth throttling example exists it would be nice to see. However I solved problem by overriding `FileSender`'s `resumeProducing` — dan.ch, Oct 29 '12 at 05:21
Regardless of what else is going on, it sounds like you've discovered a bug. You should not be getting an `AttributeError` out of ThrottlingProtocol regardless of what you do. Worst case, the exception message should be informative, but I suspect the error just shouldn't be happening. — Glyph, Jan 15 '13 at 20:54

egbutter · Answer 1 · 2013-01-14T19:54:42.327

1

Every producer needs (eventually) to be registered with whatever you want to consume the data. I don't see registration happening anywhere here. Maybe that is the issue you are having?

Twisted has been used on some big-time projects like Friendster, but all the callbacks do not sit well with the usual way I write in python (and I have some experience with functional programming). I switched to gevent.

If you are working with gevent libraries, many of the details (callbacks/generators that provide the asynchronous functionality) are abstracted out, so that you can typically get away with just monkey patching your code and writing it in the usual object-oriented style you are used to. If you are working on a project with anyone unfamiliar with a callback-heavy language like js/lisp, I bet they will appreciate gevent over twisted.

edited Jan 14 '13 at 19:54

answered Nov 30 '12 at 14:51

egbutter

810
11
21

This answer would be better (worthy of an upvote) without the pointless sniping. You don't even mention how gevent deals with flow control. – Glyph Dec 31 '12 at 08:28
@Glyph really sorry about that -- hopefully I have removed the snarkiness. Tons of respect for the (brilliant) people who developed Twisted -- I meant no offense. Perhaps just some residual scars from my bad experience. :) – egbutter Dec 31 '12 at 16:31
Thanks for altering it - but you still don't explain how gevent actually deals with the specific issue of flow control :). – Glyph Dec 31 '12 at 21:45
Haha well I do not quite understand what you want ... It is hard to explain flow control with gevent works when the principle behind gevent is to be agnostic of those details -- as opposed to a callback-centered tool like Twisted. – egbutter Jan 02 '13 at 20:35
It's not "agnostic" to flow control. It suspends execution of your microthread in the send() call; it's not any more "abstracted" than Twisted's version of flow control. If anything, it's less abstracted, since in Twisted at least there's an explicit abstraction object. – Glyph Jan 14 '13 at 19:37
@Glyph perhaps gevent is as low-level for a library developer e.g., library like [this](https://github.com/jgelens/gevent-websocket). But someone like me using those libraries to do something like [this](https://github.com/mmerickel/tictactoe) does not need to worry about the logic of `send()`'s. That makes it much easier for me to pick up and introduce async functionality. Perhaps the same is true in Twisted, just not in my limited experience. – egbutter Jan 14 '13 at 19:51

score 1 · Answer 2 · edited May 23 '17 at 11:59

1

As egbutter said, you have to register a producer. So instead of this:

tsite.protocol.producer = producer

you have to call a registerProducer method explicitly:

tsite.protocol.registerProducer( ... )

or, if you're using FileSender as a producer, call its beginFileTransfer method, in our case:

file_to_send = open( ... )
producer.beginFileTransfer(file_to_send, tsite.protocol)

edited May 23 '17 at 11:59

Community

1
1

answered Sep 20 '13 at 11:09

Oleksandr Fedorov

1,213
10
17

Bandwidth throttling using Twisted

2 Answers2