0

I've encountered this strange problem with opening/closing files in python. I am trying to do the same thing in python that i was doing successfully in matlab, and i am getting a problem communicating with some software through text files. I've come up with a strange workaround to solve the problem, but i dont understand why it works.

I have software that communicates with a some lab equipment. To communicate with this software, i write a file ('wavefile.txt') to a specific folder, containing parameters to send to the device. I then write another file named 'request.txt' containing the location of this first file ('wavefile.txt') which contains the parameters to send to the device. The software is constantly checking this folder to find the file named 'request.txt' and once it finds it, it will read the parameters in the file which is specified by the text in 'request.txt' and then delete 'request.txt'. The software/equipment developer instructs to give a 50 ms second delay before closing the 'request.txt' file.

original matlab code that works:

home = cd;
cd \\CREOL-FAST-01\data
fileID = fopen('request.txt', 'wt');
proj = 'C:\\dazzler\\data\\wavefile.txt';
fprintf(fileID, proj);
pause(0.05);
fclose('all');
cd(home);

original python code that does not work:

home = os.getcwd()
os.chdir(r'\\CREOL-FAST-01\data')
with open('request.txt', 'w') as file:
    proj = r'C:\dazzler\data\wavefile.txt'
    file.write(proj)
    time.sleep(0.05)
os.chdir(home)

Every time the device program reads the 'request.txt' when its working with matlab, it deletes it immediately after matlab closes it. When i run that code with python it works SOMETIMES, maybe 1 in every 5 tries will be successful and the parameters are sent. The 'request.txt' file is always deleted with the python code above,but the parameters i've input are clearly not sent to my lab device. My guess is that when i write the file in python, the device program is able to read it before python writes the text to it, so its just opening the blank file, not applying any parameters, and then deleting it.

My workaround in python:

home = os.getcwd()
os.chdir(r'\\CREOL-FAST-01\data')
fileh = open('request.txt', 'w+')
proj = r'C:\dazzler\data\wavefile.txt'
fileh.write(proj)
time.sleep(0.05)
print(fileh.read())
time.sleep(0.05)
fileh.close()

This method in python seems to work 100% of the time. I open the file in w+ mode, and using fileh.read() is absolutely necessary. if i delete that line and still include the extra sleeptime, it will again work about 1 in 5 tries. This seems really strange to me. Any explanation, or better solutions?

zonzon510
  • 163
  • 13
  • Did you try removing the pause, and closing the file right after you write to it? My guess is that Python doesn’t flush until you close the file, whereas MATLAB flushes right away. I don’t understand the point of the pause anyway. – Cris Luengo Jun 26 '18 at 00:00
  • 2
    Are you able to explain the reason for the pause in your code? Try to explain what your code is fundamentally trying to accomplish instead of how your current approach is not working. – Evan Benn Jun 26 '18 at 00:32
  • “The software/equipment developer instructs to give a 50 ms second delay before closing the 'request.txt' file.” I’m thinking now that maybe you misinterpreted this bit (manuals are often poorly written). Maybe the device waits 50 ms in between seeing the new file and reading and deleting it, so as to give the writer time to finish writing it (“close the file within 50 ms of opening it” rather than “close the file 50 ms after opening it”). I’m going off of what makes sense here. Waiting 50 ms makes no sense to me, I don’t understand the purpose. – Cris Luengo Jun 26 '18 at 00:46
  • It is a bizarre interface, but I am sure there are worse :(. I would recommend writing to a tmpfile and renaming into place, which will be atomic and avoid any buffering issues – Evan Benn Jun 28 '18 at 01:44

3 Answers3

2

My guess (which could be wrong) is that the file is being read before it is completely flushed. I would try using the flush() method after the write to make sure that the complete data is written to the file. You might also need the os.fsync() method to make sure the data is flushed properly. Try something like this: 

home = os.getcwd()
os.chdir(r'\\CREOL-FAST-01\data')
with open('request.txt', 'w') as file:
   proj = r'C:\dazzler\data\wavefile.txt'
   file.write(proj)
   file.flush()
   os.fsync()
   time.sleep(0.05)
os.chdir(home)
razdi
  • 1,388
  • 15
  • 21
2

Not knowing any details about the particular equipment and other software you are using it's hard to say. One guess is the difference in buffering on write calls.

From this blog post on Matlab's fwrite: "The default behavior for fprintf and fwrite is to flush the file buffer after each call to either of these functions"

Whereas for Python's open:

When no buffering argument is given, the default buffering policy works as follows:

Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying device’s “block size” and falling back on io.DEFAULT_BUFFER_SIZE. On many systems, the buffer will typically be 4096 or 8192 bytes long.

“Interactive” text files (files for which isatty() returns True) use line buffering. Other text files use the policy described above for binary files.

To test this guess change:

with open('request.txt', 'w') as file:
    proj = r'C:\dazzler\data\wavefile.txt'

to:

with open('request.txt', 'w', buffer=1) as file:
    proj = 'C:\\dazzler\\data\\wavefile.txt\n'
chf2117
  • 166
  • 9
0

The problem is probably that you are doing the delay while the file is still open and thus not written to disk. Try something like this:

home = os.getcwd()
os.chdir(r'\\CREOL-FAST-01\data')
with open('request.txt', 'w') as file:
    proj = r'C:\dazzler\data\wavefile.txt'
    file.write(proj)
time.sleep(0.05)
os.chdir(home)

The only difference here is that the sleep is done after the file is closed (the file is closed when the with block ends), and thus the delay doesn't happen until after the text is written to disk.

To put it in words, what you are doing is:

  1. Open (and create) file
  2. Write text to a buffer (in memory, not on disk)
  3. Wait 50 ms
  4. Close (and write) the file

What you want to do is:

  1. Open (and create) file
  2. Write text to a buffer (in memory, not on disk)
  3. Close (and write) the file
  4. Wait 50 ms

So what you end up with is a period of at least 50 ms where the text file has been created, but where there is nothing in it because the text is sitting in your computer memory not on disk.

To be honest, I would put as little in the with block as possible, to avoid issues like this. So I would write it like so:

home = os.getcwd()
os.chdir(r'\\CREOL-FAST-01\data')
proj = r'C:\dazzler\data\wavefile.txt'
with open('request.txt', 'w') as file:
    file.write(proj)
time.sleep(0.05)
os.chdir(home)

Also keep in mind that you also can't do the opposite: assume that no text is written until you close. For small files like this, that will probably happen. But when the file is written to disk depends on a lot of factors. When a file is closed, it is written, but it may be written before that too.

TheBlackCat
  • 9,791
  • 3
  • 24
  • 31
  • "When a file is closed, it is written, but it may be written before that too" this is not at all true, all sorts of buffering exist in python, clib, the kernel, AND the device. the path in the code example also implies some networked file system. – Evan Benn Jun 28 '18 at 01:47
  • @EvanBenn: Python explicitly calls `flush` when the file is closed. The whole point of `flush` is to force the data to be written to the file. – TheBlackCat Jun 28 '18 at 20:33
  • https://stackoverflow.com/questions/7127075/what-exactly-the-pythons-file-flush-is-doing yes that is the intent of the flush call, but computers are complicated, and there are layers in the software and hardware stack. We only deal with abstractions and apis. – Evan Benn Jun 29 '18 at 00:18