How can I implement a POSIX file descriptor in Python 3?

Question

I'd like to write a class that can behave as a bona fide file descriptor. Its .fileno() method should return a file descriptor that provides all the services a POSIX system expects.

This is my first foray into POSIX system programming, so I could be misunderstanding things quite badly.

The underlying motivation is the desire to use an in-memory Python object as the stdin or stdout kwarg to the subprocess.Popen constructor without having to rely on temporary or memory-mapped files. But I'm not interested in some clever trick that would get the job done -- I really want to have a Python implementation capable of answering all the relevant system calls.

Python "wraps" the Posix file descriptor. You have all the access you need. Please define "all the relevant system calls". Open, close, read and write are supported. Which other ones do you want? — S.Lott, Dec 30 '11 at 21:27
@S.Lott: Whatever additional calls are required to pass a StringIO object (for example) as the `stdin` or `stdout` kwarg to `subprocess.Popen` — , Dec 30 '11 at 22:34
What? If you want to use StringIO with subprocess.Popen, perhaps you should close this question and ask what you **really** want to know. — S.Lott, Dec 30 '11 at 23:09

score 2 · Answer 1 · answered Dec 30 '11 at 21:20

2

You can't. POSIX file descriptors are tracked in the operating system kernel, outside the world of Python; you can't simulate them in Python code.

answered Dec 30 '11 at 21:20

score 2 · Answer 2 · edited May 23 '17 at 11:55

If you want to have a class that can be used as a file when passed to system calls, it needs to have a fileno() that is a real OS file descriptors. One way of doing this without touching a hard-disk is to use pipes, because they have file descriptors, and the system calls can then write to these file descriptors.

I did write a class that did something using this technique for another answer. It doesn't really do what you want to do, but the technique of using pipes should be doable for you to:

import io
import logging
import os
import select
import subprocess
import time
import threading

LOG_FILENAME = 'output.log'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)

class StreamLogger(io.IOBase):
    def __init__(self, level):
        self.level = level
        self.pipe = os.pipe()
        self.thread = threading.Thread(target=self._flusher)
        self.thread.start()

    def _flusher(self):
        self._run = True
        buf = b''
        while self._run:
            for fh in select.select([self.pipe[0]], [], [], 0)[0]:
                buf += os.read(fh, 1024)
                while b'\n' in buf:
                    data, buf = buf.split(b'\n', 1)
                    self.write(data.decode())
            time.sleep(1)
        self._run = None

    def write(self, data):
        return logging.log(self.level, data)

    def fileno(self):
        return self.pipe[1]

    def close(self):
        if self._run:
            self._run = False
            while self._run is not None:
                time.sleep(1)
            os.close(self.pipe[0])
            os.close(self.pipe[1])

score 0 · Answer 3 · edited May 23 '17 at 11:56

This is my first foray into POSIX system programming, so I could be misunderstanding things quite badly.

Yep.

POSIX file descriptors are just numbers - they're not objects, so you can't override their methods. For example, 0, 1, and 2 are all [usually] valid file descriptors.

"the relevant system calls" are built in to the Linux kernel. The Linux kernel itself maintains a list that maps file descriptors to some internal kernel object (which does have methods!) but you can't insert a new file descriptor from Python. Code running in kernel space is very different from normal ("user mode") code.

Can I suggest you look at subprocess.PIPE, and either the stdout/stdin/stderr properties or the communicate() method on subprocess.Popen objects? This will let you start a subprocess, read the data that it outputs, and have full control of the data that gets sent to it. (I think this is what you're really trying to do...). If you're curious, then when you've played with this you can look at the subprocess.py source code to see how it works.

There's an example of subprocess.PIPE here.

Alternatively, if you actually want to implement a full filesystem in Python, look at FUSE, and it's Python bindings. FUSE includes a C module that runs in the kernel, and handles filesystem requests for a certain directory. It handles them by passing them to a userspace program, which could be written in Python. You can open those files from a separate Python program, to get a file descriptor to them. This is kind of complex, and probably not the best place for a beginner to start.

How can I implement a POSIX file descriptor in Python 3?

3 Answers3