9

I am facing the following problem: I'm trying to implement a simulator for supply chains. These will produce a lot of EPCIS Events (events that occur at RFID readers). These events should then be written to a csv file in order to load them into any database and run analytical algorithms on them.

The simulator is implemented using python and works fine. What I'm now trying to do is buffered writing of the events to the file in order to decrease the time, that is needed to access the disk. Browsing the python documentation I stumbled upon the io.BufferedWriter, which sounds exactly like the thing I was looking for. Anyway, I can't quite get it to work.

Here's what I did so far. I implemented my CsvWriter class, that inherits from RawIOBase and manages the file handle. When it is instantiated, it will create a BufferedWriter, handing in itself, as the raw parameter (might that already be a problem?)

class CsvWriter(AbstractWriter):

    def __init__(self, filename):    
        self.filename = filename
        self.file = self.openFile()
        self.buffer = BufferedWriter(self, settings.WRITE_THRESHOLD)

When I know want to write something I call write_buffered buffered, which looks like that:

def write_buffered(self, data_dict):    
        self.buffer.write(b';'.join(map(str, data_dict.values())) + '\n')

The actual write methods which (as I figured) needs to be implemented on the CsvWriter class itself looks like that:

def write(self, data):
        if self.file.closed:
            self.file = self.openFile()

        return self.file.write(data)

The problem is, that when I try to run the simulator, I get the following error:

IOError: raw write() returned invalid length -1 (should have been between 0 and 78)

Do any of you have a clue for me how to fix this?

philgiese
  • 623
  • 1
  • 7
  • 17
  • 1
    Ordinaly file I/O (e.g. `open`) should already be buffered (`open(...)` gives me a `Buffered{Reader,Writer}` instance), the underlying C library probably has a buffer, the OS probably has a buffer, and the disk probably has a cache. Do you really need to add anything youself? –  Jan 10 '12 at 15:50
  • Ok, but will this use as much RAM as possible before writing to disk? This is basically, what I'm trying to achieve. Keeping as much in memory as possible before I do a write out. – philgiese Jan 10 '12 at 16:03
  • 2
    You can specify the desired buffer size for a file when `open`ing it (third argument). No need to create your own buffering if you only have basic needs. Note: if you don't specify a size, it will use the system's default. [Here](http://docs.python.org/library/functions.html#open), some doc. – Ricardo Cárdenes Jan 10 '12 at 16:19

0 Answers0