5

In gevent monkey patch, I didn't see anything about default file object's operate. How can I use async file read/write in gevent based programs?

truease.com
  • 1,261
  • 2
  • 17
  • 30

3 Answers3

4

You could use gevent's fileobject.FileObjectThreadPool class available in 1.0b3: 

pip install http://gevent.googlecode.com/files/gevent-1.0b3.tar.gz#egg=gevent

Then your example would become:

#!/usr/bin/env python
import gevent
from gevent.fileobject import FileObjectThreadPool
import datetime


def hi():
    while True:
        print datetime.datetime.now(), "Hello"
        gevent.sleep( 1 )

def w():
    print "writing..."
    s = "*"*(1024*1024*1024)
    print 'about to open'
    f_raw = open( "./a.txt", "wb" )
    f = FileObjectThreadPool(f_raw, 'wb')
    f.write(s)
    f.close()
    print 'write done'

t1 = gevent.spawn(hi)
t2 = gevent.spawn(w)
ts = [t1,t2]
gevent.joinall( ts )

I see the following output with that code:

writing...
about to open
2012-08-13 13:00:27.876202 Hello
2012-08-13 13:00:28.881119 Hello
2012-08-13 13:00:29.959642 Hello
...
2012-08-13 13:00:58.010001 Hello
2012-08-13 13:00:59.010146 Hello    
2012-08-13 13:01:00.010248 Hello
write done
2012-08-13 13:01:01.469547 Hello
...
mrkhingston
  • 79
  • 7
  • Apologies - depending on what os you are using this might not work for you. I just raised a bug about this not working on ubuntu 12.04: https://code.google.com/p/gevent/issues/detail?id=150. The gevent guys are pretty good at getting bugs fixed though so it might be working on all platforms soon. – mrkhingston Aug 15 '12 at 06:38
  • FYI, Denis has now fixed issue 150 in HEAD of gevent, soon to be released as gevent-1.0b4. However, my original instructions to use FileObject were wrong - I've edited my answer to use FileObjectThreadPool. – mrkhingston Aug 31 '12 at 00:26
  • 1
    Beware that Denis has recently [changed](https://github.com/surfly/gevent/commit/184bc92992dde92477be277a4860ecde4065ee4d) the name of `FileObjectThreadPool` to `FileObjectThread` – Nikola Kotur Apr 03 '13 at 16:42
1

Just did a test, says that write a large file will block the event loop

#!/usr/bin/env python
import gevent
import datetime


def hi():
    while True:
        print datetime.datetime.now(), "Hello"
        gevent.sleep( 1 )

def w():
    print "writing..."
    s = "*"*(1024*1024*1024)
    f = open( "e:/a.txt", "wb" )
    f.write(s)
    f.close()

t1 = gevent.spawn(hi)
t2 = gevent.spawn(w)
ts = [t1,t2]
gevent.joinall( ts )

the result is this:

e:\zPython\zTest>gevent.write.large.file.py
writing...  # wait a long time here
write done.
2012-07-16 09:53:23.784000 Hello
2012-07-16 09:53:24.786000 Hello
2012-07-16 09:53:25.788000 Hello
truease.com
  • 1,261
  • 2
  • 17
  • 30
0

You can use a threadpool (starting with gevent 1.0):

>>> import gevent.threadpool
>>> pool = gevent.threadpool.ThreadPool(5)
>>> pool.apply(w)
Denis
  • 3,760
  • 25
  • 22
  • yes, I can use threadpool. But my question is not focus on gevent usage. I want to found a way to do async file io in both windows and linux boxes. Of course with python language, and work together with gevent. – truease.com Jul 16 '12 at 08:48