6

I want to read a file, but without any lock on it.

with open(source, "rb") as infile:
            data = infile.read()

Can the code above lock the source file?

This source file can be updated at any time with new rows (during my script running for example). I think not because it is only in reading mode ("rb"). But I found that we can use Windows API to read it without lock. I did not find an simple answer for my question.

My script runs locally but the source file and the script/software which appends changes on it are not (network drive).

Jason Aller
  • 3,541
  • 28
  • 38
  • 38

3 Answers3

6

Opening a file does not put a lock on it. In fact, if you needed to ensure that separate processes did not access a file simultaneously, all these processes would have to to cooperatively take special steps to ensure that only a single process accessed the file at one time (see Locking a file in Python). This can also be demonstrated by the following small program that purposely takes its time in reading a file to give another process (namely me with a text editor) a chance to append some data to the end of the file while the program is running. This program reads and outputs the file one byte at a time pausing .1 seconds between each read. During the running of the program I added some additional text to the end of the file and the program printed the additional text:

import time


with open('test.txt', "rb") as infile:
    while True:
        data = infile.read(1)
        if data == b'':
            break
        time.sleep(.1)
        print(data.decode('ascii'), end='', flush=True)

You can read your file in pieces and then join these pieces together if you need one single byte string. But this will not be as memory efficient as reading the file with a single read:

BLOCKSIZE = 64*1024 # or some other value depending on the file size
with open(source, "rb") as infile:
    blocks = []
    while True:
        data = infile.read(BLOCKSIZE)
        if data == b'':
            break
        blocks.append(data)
# if you need the data in one piece (otherwise the pieces are in blocks):
data = b''.join(blocks)
Booboo
  • 38,656
  • 3
  • 37
  • 60
3

One alternative is to make a copy of the file temporarily and read the copy. You can use the shutil package for such a task:

import os
import time
from shutil import copyfile

def read_file_non_blocking(file):
  temp_file = f"{filename}-{time.time()}"    # Stores it in the local directory
  copyfile(file, temp_file)
  with open(temp_file, 'r') as my_file:
      # Do Something cool
      my_file.close()
      os.remove(temp_file)
Lawrence Bird
  • 197
  • 9
  • 1
    My script do an incremental copy with an offset because the file is huge, I can't make a copy –  Jan 15 '21 at 10:46
  • 1
    This thread might help you instead then: https://stackoverflow.com/questions/39948588/non-blocking-file-read?rq=1 – Lawrence Bird Jan 15 '21 at 10:52
  • Ok I see there is the concurrent.futures module. But the changes are made by another script/software (on another machine). Only my script is running locally. Is this solution works in my case ? –  Jan 15 '21 at 11:03
0

Windows is weird in how it handles files if you, like myself, are used to Posix style file handling. I have run into this issue numerous times and I have been luck enough to avoid solving it. However in this case, if I had to solve it, I would look at the flags that can passed to os.open and see if any of those can disable to locking.

https://docs.python.org/3/library/os.html#os.open

I would do a little testing but I don't have a non-production critical Windows workstation to test on.

TimDaMan
  • 21
  • 2