Non-blocking file read

Question

How can I read the contents of a binary or a text file in a non-blocking mode?

For binary files: when I open(filename, mode='rb'), I get an instance of io.BufferedReader. The documentation fort io.BufferedReader.read says:

Read and return size bytes, or if size is not given or negative, until EOF or if the read call would block in non-blocking mode.

Obviously a straightforward open(filename, 'rb').read() is in a blocking mode. To my surprise, I could not find an explanation anywhere in the io docs of how to choose the non-blocking mode.

For text files: when I open(filename, mode='rt'), I get io.TextIOWrapper. I assume the relevant docs are those for read in its base class, io.TextIOBase; and according to those docs, there seems no way to do non-blocking read at all:

Read and return at most size characters from the stream as a single str. If size is negative or None, reads until EOF.

Not sure if you need to do this at a lower level using `os.O_NONBLOCK` — Padraic Cunningham, Oct 09 '16 at 21:29
please tell me why you downvoted my answer? Is it irrelevant? — Amin Etesamian, Oct 09 '16 at 22:05
@Amin Etesamian I didn't down vote and I'm not sure why someone else who read it didn't like it. It seems relevant, although I don't know aiofiles library. — max, Oct 09 '16 at 23:41
@max I used it in an async file serving service. It works quite well. — Amin Etesamian, Oct 09 '16 at 23:51

score 11 · Accepted Answer · answered Oct 09 '16 at 21:36

File operations are blocking. There is no non-blocking mode.

But you can create a thread which reads the file in the background. In Python 3, concurrent.futures module can be useful here.

from concurrent.futures import ThreadPoolExecutor

def read_file(filename):
    with open(filename, 'rb') as f:
        return f.read()

executor = concurrent.futures.ThreadPoolExecutor(1)
future_file = executor.submit(read_file, 'C:\\Temp\\mocky.py')

# continue with other work

# later:

if future_file.done():
    file_contents = future_file.result()

Or, if you need a callback to be called when the operation is done:

def on_file_reading_finished(future_file):
    print(future_file.result())

future_file = executor.submit(read_file, 'C:\\Temp\\mocky.py')
future_file.add_done_callback(on_file_reading_finished)

# continue with other code while the file is loading...

So the reference to the non-blocking mode in the documentation applies only to non-file streams? Is there a reason why file reads cannot be non-blocking? — max, Oct 09 '16 at 21:53
This solution doesn't appear to provide for any way to interrupt blocking reads to close the file, as is easy with nonblocking access. This mgiht be possible by sending a signal to the process, or maybe there is some other way to cancel a thread. I'm guessing that python file objects are designed to handle a nonblocking underlying file descriptor, which could likely be acquired via the os module. — fuzzyTew, Feb 23 '22 at 12:22

score 11 · Answer 2 · answered Feb 28 '21 at 15:21

Python does support non-blocking reads, at least on Unix type systems, by setting the O_NONBLOCK flag. In Python 3.5+, there is the os.set_blocking() function which makes this easier:

import os
f = open(filename, 'rb')
os.set_blocking(f.fileno(), False)
f.read()  # This will be non-blocking.

However, as zvone's answer notes, this doesn't necessarily work on actual disk files. This isn't a Python thing though, but an OS limitation. As the Linux open(2) man page states:

Note that this flag has no effect for regular files and block devices; that is, I/O operations will (briefly) block when device activity is required, regardless of whether O_NONBLOCK is set.

But it does suggest this may be implemented in the future:

Since O_NONBLOCK semantics might eventually be implemented, applications should not depend upon blocking behavior when specifying this flag for regular files and block devices.

If os.set_blocking is not available for you (python < 3.5), try ```fcntl.fcntl(f.fileno(), fcntl.F_SETFL, flag | os.O_NONBLOCK)``` — BeardOverflow, Jun 28 '21 at 11:17
Does the open() have to specify the "read binary" option (rb)? I'm trying it without the binary option and it seems to work. — Ben Slade, Apr 04 '23 at 21:21

Amin Etesamian · Answer 3 · 2022-02-24T15:58:55.777

6

I suggest using aiofiles - a library for handling local disk files in asyncio applications.

import aiofiles

async def read_without_blocking():
    f = await aiofiles.open('filename', mode='r')
    try:
        contents = await f.read()
    finally:
        await f.close()

edited Feb 24 '22 at 15:58

answered Oct 09 '16 at 21:19

Amin Etesamian

3,363
5
27
50

3

Not really clear what context your yield statements are suppose to go into, could you provide a more functional example? – Tadhg McDonald-Jensen May 30 '19 at 01:47
How do I call the async function? I get an error that the function is not awaited. How do I call an async function from a sync function? – Shailesh May 23 '20 at 13:17
@Yankee https://stackoverflow.com/questions/55647753/call-async-function-from-sync-function-while-the-synchronous-function-continues – Amin Etesamian May 23 '20 at 22:12
@Juggernaut I'm sorry that still doesn't make it clear how to use the above code. Could you please edit your code to include a sample execution – Shailesh May 24 '20 at 13:55
This is likely the best answer here. Any quirks one encounters can be abstracted away into a maintained library. The downside is that asyncio involves a different python coding paradigm than normal synchronous code. It takes some intro reading of the python asyncio module, to launch a run loop and write async functions to use the library. The answer might need to be updated for modern code, I'm not sure 'yield from' is still an asyncio thing, could be wrong. – fuzzyTew Feb 23 '22 at 12:25

Non-blocking file read

3 Answers3

Linked