Check if a File is written or in use by another process

Question

I try to find a solution for the following problems but can't find any good solution. I have a folder with subfolder and files in it. Some of the files may be in use by another process (the other process is writing data to the data (a .mdf file)).

I simply want to check if the files are in use or not.

Structure:

A_Folder
  Setup
  Data1
    .mdf-file1
    .mdf-file2
  Data2
  Data3
  Evaluation

something like:

def file_in_use():
    *your solution*

for file in folder:
    if file_in_use(file):
        print("file in use")
        break

I'm Using Win10, PyCharm and a venv.

I tried so far form other "solutions":

psutil (works, but is too slow)
open(), os.rename - won't work for me
subprocess wont work either -cant find my filename: using the method from Amit Gupta from my link down below, file looks like this: "C:\Data\S_t_h\S-t-h\H001.mdf"

basically I tried everything from this question:

Check if a file is not open nor being used by another process

from subprocess import check_output, Popen, PIPE

src = r"C:\Data\S_t_h\S-t-h\H001.mdf"
files_in_use = False


def file_in_use(src):

    try:
        lsout = Popen(['lsof', src], stdout=PIPE, shell=False)
        check_output(["grep", src], stdin=lsout.stdout, shell=False)
    except:
        return False

return True


if file_in_use(src):
    files_in_use = True

and im getting: FileNotFoundError: [WinError 2] The system cannot find the file specified

this link suggesting setting

winerror-2-the-system-cannot-find-the-file-specified-python

shell=True

Im getting "lsof" and "grep" cant be found or are wrong now.

Here the psutil method that works for me, but is too slow (~10 Seconds)

import psutil

src = r"C:\Data\S_t_h\S-t-h\H001.mdf"

def has_handle(src):
    for proc in psutil.process_iter():
        try:
            for item in proc.open_files():
                if src == item.path:
                    return True
        except Exception:
            pass

    return False


 print(has_handle(src))

My Solution:

Sorry for the delayed answer. It simply worked with:

    try:
        os.rename(src, src)
        return False
    except OSError:    # file is in use
        return True

I made it more complicated than it actually was i guess. But thank you guys anyway for your feedback and critizism.

You may want to look at this: https://stackoverflow.com/questions/17321930/check-if-file-is-open-with-lsof — Jim Black, Feb 11 '21 at 16:45
Hi @JimBlack, thanks, but theres no lsof library i can try in pycharm (im not using a linux server, sorry for that) but lsof is in the psutil library which is, for me, too slow (takes around 20 seconds for every file to check) — Hamholder, Feb 11 '21 at 17:05
The better answer is to rewrite the _other_ program so it follows best practices: Write to a temporary file (on the same filesystem as the destination), not the final name, and rewrite it to the "real" name only when it's completely done being written. That way it _never_ exists under the final name at all, except in fully-written state. You say "won't work for me", but you aren't saying anything about _why_ or _how_ it doesn't work, or giving us a chance to figure out how to fix it so it _will_ work. — Charles Duffy, Feb 11 '21 at 18:57
...similarly, saying `subprocess` "can't find my filename" doesn't really make sense. I'd need to see _how_ you're trying to use `subprocess` and exactly what error you get to explain it; probably, that would be a solvable problem. — Charles Duffy, Feb 11 '21 at 18:58
...that said, Windows -- specifically -- will already make attempts to open a file for write if another program is already writing to it fail; so if Windows is the only platform you need to support, all you need is some exception handling. — Charles Duffy, Feb 11 '21 at 18:59
@CharlesDuffy yeah, you're right about rewriting the other program.. will do if i have the time for it. the subprocess says it cant find the file, maybe im using the wrong // / \ thingy for the file str or it has sth to do with the venv. the error im getting is a FileNotFoundError when using this method from Amit Gupta in my posted link. My File String looks like this: C:\\Data\\S_t_h\\S-t-h\\H001.mdf — Hamholder, Feb 11 '21 at 19:05
I'd really need a [mre], not just a comment, to be able to dig into that. — Charles Duffy, Feb 11 '21 at 19:16
It's strongly recommended to make your question free of spelling error -- that shows that you don't write your question sloppily. — user202729, Feb 12 '21 at 07:09
What does "too slow" mean? Do you have 1000 files in that folder and you need to check if any of them are in use? — user202729, Feb 12 '21 at 07:10
@user202729 OP specified around 20 seconds per file checked, which I think can be reasonably considered "too slow" for most applications. — Layne Bernardo, Feb 12 '21 at 07:53
@LayneBernardo I think that statement means "20 seconds for all the files in the folder". Because 20 seconds for a single file is impossibly slow (except perhaps on a bad network connection). — user202729, Feb 12 '21 at 07:58
@CharlesDuffy I just checked, and this is actually not true. Probably has to do with Python buffering writes, but you can definitely open the same file with write access in two different Python scripts at the same time on Windows 10. The contents will be whichever script calls file.close() first, as you would expect. — Layne Bernardo, Feb 12 '21 at 07:58
If you were on Linux, you could easily protect the file using flock(), but since you're on Windows I don't think there's any native file locking solution. You could probably find a Python file locking library, or just implement your own - it's really easy to implement simple locking. One way is to just create and delete a lockfile. If the lockfile exists, the file is being written to or something went wrong and it's in an unknown state. — Layne Bernardo, Feb 12 '21 at 08:01
@user202729 im sorry for my english, its not my main language and i tried my best and its not meant in any way to be sloppy (mean) — Hamholder, Feb 12 '21 at 08:21
Did you mean to say 20 seconds per file, or 20 seconds for the whole directory? — user202729, Feb 12 '21 at 08:23
@user202729 its 20 seconds per file with the psutil method shown in the first link — Hamholder, Feb 12 '21 at 08:43
@LayneBernardo Interesting. Python needs to be using newer APIs (that aren't available at all in ancient XP-era Windows releases) to do that; moreover, it's a breaking change for applications that assume the old semantics, so I'm surprised it went in (unless it happened as if 3.0, which would make a lot of sense). Anyhow, does adding `x` to the open flags fix it? — Charles Duffy, Feb 12 '21 at 15:37
@JimBlack Im trying to find out if a .mdf is being written constantly of another process. — Hamholder, Feb 15 '21 at 07:11
@Hamholder, then the file's size and modification date/time stamp will be constantly changing. Those are pretty easy to check using stat(). Is this a security check? Are you trying to do something with the files? — Jim Black, Feb 15 '21 at 18:49

Check if a File is written or in use by another process

My Solution:

0 Answers0