75

I my application, i have below requests: 1. There has one thread will regularly record some logs in file. The log file will be rollovered in certain interval. for keeping the log files small. 2. There has another thread also will regularly to process these log files. ex: Move the log files to other place, parse the log's content to generate some log reports.

But, there has a condition is the second thread can not process the log file that's using to record the log. in code side, the pseudocode similars like below:

#code in second thread to process the log files
for logFile in os.listdir(logFolder):
     if not file_is_open(logFile) or file_is_use(logFile):
          ProcessLogFile(logFile) # move log file to other place, and generate log report....

So, how do i check is a file is already open or is used by other process? I did some research in internet. And have some results:

try:
   myfile = open(filename, "r+") # or "a+", whatever you need
except IOError:
    print "Could not open file! Please close Excel!"

I tried this code, but it doesn't work, no matter i use "r+" or "a+" flag

try:
   os.remove(filename) # try to remove it directly
except OSError as e:
    if e.errno == errno.ENOENT: # file doesn't exist
        break

This code can work, but it can not reach my request, since i don't want to delete the file to check if it is open.

Aitor
  • 88
  • 1
  • 9
zengwke
  • 761
  • 1
  • 6
  • 6
  • Have you tried to change `os.remove` to `ProcessLogFile` inside the last `try` block? Maybe tweak the error number: there are `EBUSY` and [others](http://docs.python.org/library/errno.html) to try. – Lev Levitsky Jun 20 '12 at 07:53
  • 1
    You may wanna read this question http://stackoverflow.com/questions/2023608/check-what-files-are-open-in-python and especially this http://stackoverflow.com/a/7142094/546873 answer – Nicola Coretti Jun 20 '12 at 08:09
  • How to do the similar things at Windows platform to list open files. – zengwke Jun 20 '12 at 08:20
  • related: https://stackoverflow.com/q/589407 – djvg Mar 31 '22 at 10:06

9 Answers9

62

An issue with trying to find out if a file is being used by another process is the possibility of a race condition. You could check a file, decide that it is not in use, then just before you open it another process (or thread) leaps in and grabs it (or even deletes it).

Ok, let's say you decide to live with that possibility and hope it does not occur. To check files in use by other processes is operating system dependant.

On Linux it is fairly easy, just iterate through the PIDs in /proc. Here is a generator that iterates over files in use for a specific PID:

def iterate_fds(pid):
    dir = '/proc/'+str(pid)+'/fd'
    if not os.access(dir,os.R_OK|os.X_OK): return

    for fds in os.listdir(dir):
        for fd in fds:
            full_name = os.path.join(dir, fd)
            try:
                file = os.readlink(full_name)
                if file == '/dev/null' or \
                  re.match(r'pipe:\[\d+\]',file) or \
                  re.match(r'socket:\[\d+\]',file):
                    file = None
            except OSError as err:
                if err.errno == 2:     
                    file = None
                else:
                    raise(err)

            yield (fd,file)

On Windows it is not quite so straightforward, the APIs are not published. There is a sysinternals tool (handle.exe) that can be used, but I recommend the PyPi module psutil, which is portable (i.e., it runs on Linux as well, and probably on other OS):

import psutil

for proc in psutil.process_iter():
    try:
        # this returns the list of opened files by the current process
        flist = proc.open_files()
        if flist:
            print(proc.pid,proc.name)
            for nt in flist:
                print("\t",nt.path)

    # This catches a race condition where a process ends
    # before we can examine its files    
    except psutil.NoSuchProcess as err:
        print("****",err)
J.G
  • 190
  • 3
  • 13
cdarke
  • 42,728
  • 8
  • 80
  • 84
  • Thanks for your answers. But, sorry i can not try to install the psutil package. Since the limitation of application framework. I can not include the other thirdparty packages. Is there any way can do this by using pure python2.4? – zengwke Jun 20 '12 at 08:38
  • Not using the standard library, no. Another alternative is to write it yourself in C or using ctypes - a lot of work – cdarke Jun 20 '12 at 09:11
  • 3
    Very good, but in your Linux example, I suggest using errno.ENOENT instead of the value 2. – kmarsh Nov 03 '15 at 16:56
  • @zengwke: why not install `psutil` under your account, i.e., `~/` ? – 0 _ Feb 14 '16 at 23:58
  • 4
    This worked for me but I also had to catch `psutil.AccessDenied` exceptions – FHTMitchell Jul 09 '19 at 09:51
  • On Windows, you'd usually open the file exclusively - it either succeeds, denying others any access while you hold it, or it fails because someone else has a (non-sharing) handle open. Avoids the race coniditon nicely. – peterchen Jun 30 '23 at 15:09
40

I like Daniel's answer, but for Windows users, I realized that it's safer and simpler to rename the file to the name it already has. That solves the problems brought up in the comments to his answer. Here's the code:

import os

f = 'C:/test.xlsx'
if os.path.exists(f):
    try:
        os.rename(f, f)
        print 'Access on file "' + f +'" is available!'
    except OSError as e:
        print 'Access-error on file "' + f + '"! \n' + str(e)
MarredCheese
  • 17,541
  • 8
  • 92
  • 91
  • 9
    I'm pretty sure this won't work on non-Windows OS's (my Linux system readily let me rename a database file I had open in another process). – Big_Al_Tx Jan 01 '18 at 20:06
37

You can check if a file has a handle on it using the next function (remember to pass the full path to that file):

import psutil

def has_handle(fpath):
    for proc in psutil.process_iter():
        try:
            for item in proc.open_files():
                if fpath == item.path:
                    return True
        except Exception:
            pass

    return False
Tavy
  • 861
  • 11
  • 15
  • 1
    Really nice ! Thanks – ZHAJOR Oct 17 '17 at 22:02
  • 2
    Very nice solution. Is this cross platform? It works well for me on linux, how about windows? – David Parks Apr 06 '19 at 00:35
  • 1
    while I vim a file, but it returned False. Is there something wrong? – DennisLi Sep 24 '19 at 08:52
  • 2
    @DennisLi Same happened to me. Vim seems to use `.swp` files that it keeps in a vim directory in `~/.config`. The original file is not keps open by Vim (well, Neovim in my case). – C14L Feb 13 '20 at 14:56
  • 1
    This did help wait for a file copy transfer to finish, before posting the file. Thanks, I had tried comparing size, modification times, etc, and none of that worked. – msoutopico Feb 17 '20 at 18:33
  • Beware, methods `absolute()` and `resolve()` of `pathlib.Path` objects do not return a `str` as required here! – Suuuehgi May 17 '20 at 19:16
  • 1
    This didn't work for me, for some reason it keeps on returning False (Ubuntu 18.04, Python 3.7) – YTZ May 22 '20 at 20:57
  • @YTZ Did you call the method with the full path? Are you sure that the file you are checking is opened by another process (to read/write to it)? Maybe you do not have permissions. Try printing the exceptions and the PID associated, and maybe you'll find out the problem – Tavy May 23 '20 at 13:54
  • this one works really good. In my script I am processing data that are written by another process, so I have to make sure the file does not have a handle. thanks! – xaratustra May 01 '21 at 05:20
  • 1
    What exception is thrown here? `IOException`? – Gulzar Oct 05 '22 at 07:10
  • 1
    This fails with psutil.AccessDenied exception – Matti Jokipii Jun 02 '23 at 18:16
9

I know I'm late to the party but I also had this problem and I used the lsof command to solve it (which I think is new from the approaches mentioned above). With lsof we can basically check for the processes that are using this particular file. Here is how I did it:

from subprocess import check_output,Popen, PIPE
try:
   lsout=Popen(['lsof',filename],stdout=PIPE, shell=False)
   check_output(["grep",filename], stdin=lsout.stdout, shell=False)
except:
   #check_output will throw an exception here if it won't find any process using that file

just write your log processing code in the except part and you are good to go.

Amit G
  • 186
  • 1
  • 11
3

You can use inotify to watch for activity in file system. You can watch for file close events, indicating that a roll-over has happened. You should also add additional condition on file-size. Make sure you filter out file close events from the second thread.

Community
  • 1
  • 1
tuxuday
  • 2,977
  • 17
  • 18
3

Instead on using os.remove() you may use the following workaround on Windows:

import os

file = "D:\\temp\\test.pdf"
if os.path.exists(file):
    try:
        os.rename(file,file+"_")
        print "Access on file \"" + str(file) +"\" is available!"
        os.rename(file+"_",file)
    except OSError as e:
        message = "Access-error on file \"" + str(file) + "\"!!! \n" + str(e)
        print message
Daniel
  • 31
  • 1
  • 9
    Race condition here. If the user interrupts the program (ctrl-c) after the first rename then the filename will not be restored and the user will be unaware of this condition. At a minimum you should pair the two rename operations together. The print should go after. This minimizes the window of danger. os.rename(---); os.rename(---); print "Access ---" You should also catch KeyboardInterrupt and SystemExit exceptions so you can attempt to restore the filename before the application quits. – Noah Spurrier Jun 06 '15 at 21:25
3

A slightly more polished version of one of the answers from above.

from pathlib import Path


def is_file_in_use(file_path):
    path = Path(file_path)
    
    if not path.exists():
        raise FileNotFoundError
    
    try:
        path.rename(path)
    except PermissionError:
        return True
    else:
        return False
Razbi
  • 31
  • 1
  • 6
1

On Windows, you can also directly retrieve the information by leveraging on the NTDLL/KERNEL32 Windows API. The following code returns a list of PIDs, in case the file is still opened/used by a process (including your own, if you have an open handle on the file):

import ctypes
from ctypes import wintypes

path = r"C:\temp\test.txt"

# -----------------------------------------------------------------------------
# generic strings and constants
# -----------------------------------------------------------------------------

ntdll = ctypes.WinDLL('ntdll')
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

NTSTATUS = wintypes.LONG

INVALID_HANDLE_VALUE = wintypes.HANDLE(-1).value
FILE_READ_ATTRIBUTES = 0x80
FILE_SHARE_READ = 1
OPEN_EXISTING = 3
FILE_FLAG_BACKUP_SEMANTICS = 0x02000000

FILE_INFORMATION_CLASS = wintypes.ULONG
FileProcessIdsUsingFileInformation = 47

LPSECURITY_ATTRIBUTES = wintypes.LPVOID
ULONG_PTR = wintypes.WPARAM


# -----------------------------------------------------------------------------
# create handle on concerned file with dwDesiredAccess == FILE_READ_ATTRIBUTES
# -----------------------------------------------------------------------------

kernel32.CreateFileW.restype = wintypes.HANDLE
kernel32.CreateFileW.argtypes = (
    wintypes.LPCWSTR,      # In     lpFileName
    wintypes.DWORD,        # In     dwDesiredAccess
    wintypes.DWORD,        # In     dwShareMode
    LPSECURITY_ATTRIBUTES,  # In_opt lpSecurityAttributes
    wintypes.DWORD,        # In     dwCreationDisposition
    wintypes.DWORD,        # In     dwFlagsAndAttributes
    wintypes.HANDLE)       # In_opt hTemplateFile
hFile = kernel32.CreateFileW(
    path, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, None, OPEN_EXISTING,
    FILE_FLAG_BACKUP_SEMANTICS, None)
if hFile == INVALID_HANDLE_VALUE:
    raise ctypes.WinError(ctypes.get_last_error())


# -----------------------------------------------------------------------------
# prepare data types for system call
# -----------------------------------------------------------------------------

class IO_STATUS_BLOCK(ctypes.Structure):
    class _STATUS(ctypes.Union):
        _fields_ = (('Status', NTSTATUS),
                    ('Pointer', wintypes.LPVOID))
    _anonymous_ = '_Status',
    _fields_ = (('_Status', _STATUS),
                ('Information', ULONG_PTR))


iosb = IO_STATUS_BLOCK()


class FILE_PROCESS_IDS_USING_FILE_INFORMATION(ctypes.Structure):
    _fields_ = (('NumberOfProcessIdsInList', wintypes.LARGE_INTEGER),
                ('ProcessIdList', wintypes.LARGE_INTEGER * 64))


info = FILE_PROCESS_IDS_USING_FILE_INFORMATION()

PIO_STATUS_BLOCK = ctypes.POINTER(IO_STATUS_BLOCK)
ntdll.NtQueryInformationFile.restype = NTSTATUS
ntdll.NtQueryInformationFile.argtypes = (
    wintypes.HANDLE,        # In  FileHandle
    PIO_STATUS_BLOCK,       # Out IoStatusBlock
    wintypes.LPVOID,        # Out FileInformation
    wintypes.ULONG,         # In  Length
    FILE_INFORMATION_CLASS)  # In  FileInformationClass

# -----------------------------------------------------------------------------
# system call to retrieve list of PIDs currently using the file
# -----------------------------------------------------------------------------
status = ntdll.NtQueryInformationFile(hFile, ctypes.byref(iosb),
                                      ctypes.byref(info),
                                      ctypes.sizeof(info),
                                      FileProcessIdsUsingFileInformation)
pidList = info.ProcessIdList[0:info.NumberOfProcessIdsInList]
print(pidList)
Robert
  • 1,357
  • 15
  • 26
0

I provided one solution. please see the following code.

def isFileinUsed(ifile):
    widlcard = "/proc/*/fd/*"
    lfds = glob.glob(widlcard)
    for fds in lfds:
        try:
            file = os.readlink(fds)
            if file == ifile:
                return True            
        except OSError as err:
            if err.errno == 2:     
                file = None
            else:
                raise(err)
    return False

You can use this function to check if a file is in used.

Note: This solution only can be used for Linux system.

Pogo Lin
  • 1,901
  • 1
  • 11
  • 7