27

I'm trying to figure out if there is a defacto pattern for file access using twisted. Lots of examples I've looked at (twisted.python.log, twisted.persisted.dirdbm, twisted.web.static) actually don't seem to worry about blocking for file access.

It seems like there should be some obvious interface, probably inheriting from abstract.FileDescriptor, that all file access should be going through it as a producer/consumer.

Have I missed something or is it just that the primary use for twisted in asynchronous programming is for networking and it hasn't really been worked out for other file descriptor operations, not worrying about the purity of non-blocking IO ?

rhettg
  • 2,453
  • 18
  • 17

5 Answers5

14

I think you are looking for the fdesc module. For more information on non-blocking I/O in Python, you can also watch this video.

n.st
  • 946
  • 12
  • 27
Bertolt
  • 995
  • 1
  • 13
  • 37
3

There is an open ticket for this in Twisted - #3983.

rlotun
  • 7,897
  • 4
  • 28
  • 23
2

After plenty of searching, trial, and error, I finally figured how to use fdesc.

from __future__ import print_function

from twisted.internet.task import react
from twisted.internet import stdio, protocol
from twisted.internet.defer import Deferred
from twisted.internet.fdesc import readFromFD, setNonBlocking


class FileReader(protocol.Protocol):
    def __init__(self, filename):
        self.f = open(filename, 'rb')

    def dataReceived(self, data):
        self.transport.write(data)

    def connectionMade(self):
        fd = self.f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, self.dataReceived)

    def connectionLost(self, reason):
        self.f.close()

def main(reactor, filename):
    stdio.StandardIO(FileReader(filename))

[Edit: I also just figured out a simpler way that doesn't require the use of a protocol]

def getFile(filename):
    with open(filename) as f:
        d = Deferred()
        fd = f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, d.callback)
        return d


def main(reactor, filename):
    d = getFile(filename)
    return d.addCallback(print)

Run either like so:

react(main, ['/path/to/file'])
reubano
  • 5,087
  • 1
  • 42
  • 41
  • 2
    In the second example, does it matter if the file is closed while the deferred hasn't fired? – Chris Feb 22 '16 at 20:48
  • 1
    Good point @Chris. Maybe it'd be better to forgoe the context manager and explicitly close the file in the callback. – reubano Feb 23 '16 at 09:23
  • This does blocking file access, though. Unix systems silently ignore `setNonBlocking` when the fd points to an ordinary filesystem file. – Nathaniel J. Smith Jun 04 '17 at 15:12
  • @NathanielJ.Smith do you have a link to any documentation stating this? – reubano Jun 06 '17 at 17:18
2

The fdesc module might be useful for asynchronously talking to a socket or pipe, but when given an fd that refers to an ordinary filesystem file, it does blocking io (and via a rather odd interface at that). For disk io, fdesc is effectively snake oil; don't use it.

As of May 2017, the only reasonable way to get async disk io in twisted is by wrapping synchronous io calls in a deferToThread.

Nathaniel J. Smith
  • 11,613
  • 4
  • 41
  • 49
-8

I'm not sure what you want to achieve. When you do logging, then Python will make sure (by the global interpreter log) that log messages from several threads go into the file one after the other.

If you're concerned about blocking IO, then the OS adds default buffers for your files (usually 4KB), and you can pass a buffer size in the open() call.

If you're concerned about something else, then please clarify your question.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820