0

Firstly I am new to Python. Now my question goes like this:

I have a call back script running in remote machine which sends some data and run a script in local machine which process that data and write to a file. Now another script of mine locally needs to process the file data one by one and delete them from the file if done. The problem is the file may be updating continuoulsy. How do i schyncronize the work so that it doesnt mess up my file. Also please suggest me if the same work can be done in some better way.

bluefoggy
  • 961
  • 1
  • 9
  • 23

2 Answers2

1

I would suggest you to look into named pipes or sockets which seem to be more suited for your purpose than a file. If it's really just between those two applications and you have control on the source code of both.

For example, on unix, you could create a pipe like (see os.mkfifo):

import os
os.mkfifo("/some/unique/path")

And then access it like a file:

dest = open("/some/unique/path", "w")  # on the sending side
src = open("/some/unique/path", "r")   # on the reading side

The data will be queued between your processes. It's a First In First Out really, but it behaves like a file (mostly).

If you cannot go for named pipes like this, I'd suggest to use IP sockets over localhost from the socket module, preferably DGRAM sockets, as you do not need to do some connection handling there. You seem to know how to do networking already.

Jonas Schäfer
  • 20,140
  • 5
  • 55
  • 69
  • But as i mentioned , the first script will keep appending some data to the file.The second script may be will read line by line and do some work and if success it should remove that data(each line) from that file. Will pipe help in this scenario. As far as i know pipe is to use for reading from one and writing from another end. – bluefoggy Jul 04 '12 at 16:33
  • Ah, now I see the point. You did not mention that deleting the line occurs only on success. Hm. In that case you could try to the following. Use the fifo for IPC and write out the lines which did _not_ work out successfully to a new file. The new file will have the same contents as the intermediate file had in your scenario, but it will be easier to manage. – Jonas Schäfer Jul 04 '12 at 16:37
  • One more question here. At some point of time the intermediate file(pipe) will become huge. How do i trim that to make it less as older data will not be needed any more . – bluefoggy Jul 04 '12 at 16:48
  • I'm not sure I understand the situation correctly. The fifo is a linear structure and there is a limit in size. This implies that if the queue is full, the writing process will block while trying to write. If you need realtime transmission and the possibility to discard old (unprocessed) data, you possibly need a different solution, maybe based on Datagram sockets and threading. – Jonas Schäfer Jul 04 '12 at 16:51
0

I would suggest using a database whose transactions allow for concurrent processing.

Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358