58

Does asyncio supports asynchronous I/O for file operations? If yes, how I can use this in Python 3.5 with async/await syntax code?

CthUlhUzzz
  • 637
  • 1
  • 5
  • 4
  • I mean, i need non-blocking file I/O functionality. But in docs i found only descriptors monitoring functions. – CthUlhUzzz Jan 10 '16 at 00:03
  • If you just want non-blocking IO you should just be able to use python's threads. Do you want async IO or non-blocking IO? http://stackoverflow.com/questions/319132/asynchronous-file-writing-possible-in-python might have some useful info. – Tom Dalton Jan 10 '16 at 00:09
  • Let's explain. I have several Tasks in my IOLoop, that are working with sockets. And I want to add one more Task, that will read data to send from a file. Synchronization will be performed by asyncio.Queue. – CthUlhUzzz Jan 10 '16 at 01:28

4 Answers4

39

Most operating systems don't support asynchronous file operations. That's why asyncio doesn't support them either.

See the asyncio wiki for further explanation.

Smi
  • 13,850
  • 9
  • 56
  • 64
Andrew Svetlov
  • 16,730
  • 8
  • 66
  • 69
  • 1
    It is. Even `node.js` uses thread pool internally for providing async file API – Andrew Svetlov Aug 01 '17 at 06:34
  • You mean they do support it? – lucid_dreamer Jun 03 '18 at 20:07
  • 8
    They *emulate* it, it makes a difference. – Andrew Svetlov Jun 04 '18 at 05:59
  • It is not achieved cooperatively, so it suffers from having to to code around pre-emptive context switches, so more inefficient all in all? – lucid_dreamer Jun 04 '18 at 12:25
  • 2
    It uses thread pool internally, not cooperative switch. Performance is quite good though. – Andrew Svetlov Jun 05 '18 at 11:24
  • 1
    This is wildly wrong. Windows supports asynchronous file I/O via the OVERLAPPED flag (with the notable exception of opening files). libuv uses this. – Anton Lahti Aug 18 '20 at 08:20
  • 1
    @AntonLahti Windows is "wildly" different from most other OS. Can you name any others that do support this? Note the answer states *most* don't support it. – Philip Couling Aug 20 '20 at 15:23
  • 1
    I wonder if there is any more recent information than from more than 6 years ago (the article literally mentions "Recent discussion on the Linux Kernel: Non-blocking buffered file read operations (__September 2014__)." as the last point of reference). I highly doubt there hasn't been any change in that time. – FrankyBoy Mar 21 '21 at 17:52
  • Nothing is changed AFAIK, sorry. – Andrew Svetlov Mar 28 '21 at 20:46
  • 1
    file descriptor (Unix/Linux) is stateful for example if you seek (to relative position) then read, and seek (to relative position) then read the results of read depends on position which is not what you expected. To implement asyncio on files there is special/different kernel routines called [kernel aio](https://kkourt.io/blog/2017/10-14-linux-aio.html) – Muayyad Alsadi Jul 12 '21 at 14:30
  • Even if the OS implementation isn't asynchronous, blocking the asyncio event loop may be undesirable. – Michael Bosworth Sep 04 '21 at 10:40
  • 1
    Not true anymore since linux has `io_uring` and windows always had async file io. – Guillaume Racicot Nov 22 '21 at 15:01
  • most meaning "windows does but i don't care?" – Erik Aronesty Dec 15 '22 at 18:52
  • @ErikAronesty most meaning most. If the OS that the vast majority of Python code is running on, much less the vast majority of Python code that cares about this level of performance in the first place, does not support async IO, then there's less of a case to implement it. – Kye Apr 07 '23 at 05:09
29

That depends on what library you use.

curio

curio offers this functionality, see https://curio.readthedocs.io/en/latest/reference.html#module-curio.file

asyncio

Update 2021: aiofile ~2 and ~3 (current) supports true asynchronous IO on Linux >= 4.18 via https://github.com/mosquito/caio and falls back to threaded implementations otherwise.

Plain asyncio doesn't, although there are 3rd party libraries, e.g. aiofiles (where synchronous file access is isolated in threads) and aiofile (note the spelling) (where synchronous file access is in threads in other circumstances than the above paragraph)

Modern operating systems do provide asynchronous file primitives, but these are varied, thus each would need own implementation. Please compare:

I suspect someone will soon rip out underlying async io from node.js and make a decent Python library, or perhaps someone already has.

Specifically for Linux, there are low-level bindings in https://pypi.org/project/liburing/

For a solid overview of asynchronous IO APIs in Linux, circa 2020, see https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/

Dima Tisnek
  • 11,241
  • 4
  • 68
  • 120
  • 6
    As far as I know, all options that you've mentioned use threads (curio, aiofiles, glib's aio implementation, and even Windows Overlapped I/O uses a thread pool under the hood). – jfs Jun 20 '17 at 22:17
  • 1
    Is there a recommendation? aiofiles as suggested by the official python asyncio docs? – lucid_dreamer Jun 04 '18 at 12:26
  • 1
    For `curio`, the canonical recommendation is `curio.file`; There doesn't seem to be a recommendation for `asyncio`, so pick whatever's popular, like `aiofiles`. – Dima Tisnek Jun 09 '18 at 07:49
  • 2
    Apparently, ``aiofile`` (not ``aiofiles``!) supports [Linux libaio asynchronous file operations since version 2.0](https://pypi.org/project/aiofile/). – MisterMiyagi Oct 12 '21 at 09:08
  • Cheers, @MisterMiyagi I've updated the answer. – Dima Tisnek Oct 14 '21 at 00:56
14

asyncio does not have support for this. However, aiofiles supports just this. Please have a look.

Z. Qui
  • 189
  • 2
  • 4
5

As per Python 3.9 this is possible to do with asyncio. https://docs.python.org/3.9/library/asyncio-task.html#asyncio.to_thread

await asyncio.to_thread(shutil.copyfile, "a", "b")
Swedgin
  • 815
  • 1
  • 11
  • 20