Python retry to open the file

Question

What is the best practice to try to open the file and retry after n seconds?

Currently, I do:

import os
from os import path
import shutil

dir_path = path.abspath(path.join("foo", "bar"))
destination_path = path.abspath(path.join("foo", "dest_dir"))

for f in dir_path:
    try:
        # try to open the file if the file isn't used any longer
        file_opened = open(f, 'r')
        # move the file on successful opening
        shutil.move(file_opened, destination_path)
        file_opened.close()
    except IOError:
        return False

So, at the moment I do not handle the exception. I think about creation of extra function to open the file and recall the function on excepth with time.sleep(n)

However, I am sure there must be something else ...

I do not use

with open(f, 'rb') as file_opened: 
    do whatever`

EDIT:

One process creates the file and I want Python process to move the file once I am sure the file writting / creation is completed. So, I have added shutil.move in the above code to show the whole situation.

EDIT:

Please find below the code I have developed to solve the problem. I ended with writing own custom solution to handle it:

import os
from os import path
import psutil
from retry import retry
import shutil
from subprocess import check_output,Popen, PIPE
import glob
import time


class FileHandler:
    def __init__(self, fn_source, dir_source):
        self.file_source_name = fn_source
        self.file_source_path = path.join(dir_source, self.file_source_name)
        self.dir_dest_path = path.join(dir_source, "test")
        self.file_dest_path = path.join(self.dir_dest_path, self.file_source_name)

    def check_file(self):
        if os.path.exists(self.file_source_path):
            try:
                os.rename(self.file_source_path, self.file_source_path)
                print("file renamed")
                return True
            except:
                print("can not rename the file..retrying")
                time.sleep(1)
                self.check_file()
        else:
            print("source file does not exist...retrying")
            time.sleep(5)
            self.check_file()

    def check_destination(self):
        if os.path.exists(self.file_source_path) and not os.path.exists(self.file_dest_path):
            return True
        elif os.path.exists(self.file_source_path) and os.path.exists(self.file_dest_path):
            try:
                print(self.file_dest_path, self.file_source_name)
                os.remove(self.file_dest_path)
                return True
            except Exception as e:
                print("can not remove the file..retrying")
                time.sleep(5)
                self.check_destination()

    def move_file(self):
        if self.check_destination():
            print(self.file_source_path)
            shutil.move(self.file_source_path, self.file_dest_path)
            print("moved", str(self.file_source_path))
            return True
        else:
            print("can not move the file..retrying")
            time.sleep(1)
            self.move_file()

    def file_ops(self):
        if self.check_file():
            self.move_file()
        else:
            print("source does not exist")
            time.sleep(1)
            self.file_ops()
        return True


def each_file_ops(fn, dir_source):
    fh = FileHandler(fn, dir_source)
    return fh.file_ops()


def main(dir_source):
    dir_files = glob.glob(path.join(dir_source, '*.txt'))
    if dir_files:
        [each_file_ops(f, dir_source) for f in dir_files]
    else:
        print("source dir is empty")
        time.sleep(1)
        main(dir_source)


if __name__ == '__main__':
    main(path.join(""))

What is the point of trying to open the file that could not be opened? — DYZ, Jun 21 '18 at 06:16
The point is that if you can not open the file right here right now because sth may be still using it than you may want to come back and check again to see if you can open the file? — mothership, Jun 21 '18 at 06:18
e.g. monster1 creates the file for you but there is no way to speak with the monster1 so instead you may want to recheck to try to open the file and see if the monster1 is gone and the file is left for you so you can play with it. — mothership, Jun 21 '18 at 06:20
The you should wait for the lock to become available, rather than keep trying. — DYZ, Jun 21 '18 at 06:20
what lock? I am not multithreading if this is what you mean (GIL?) but I may be misunderstanding you. — mothership, Jun 21 '18 at 06:21
Hello schwobaseggl . My bad. It shall be f. Sorry. I have just corrected it. I was rewritting the code from the screen instead of copy paste. — mothership, Jun 21 '18 at 06:22
What OS do you use, in the first place? The only way to acquire a file on Linux is to lock it. Windows probably uses a similar mechanism. — DYZ, Jun 21 '18 at 06:25
Since you are opening the file in the read mode there should not be any exception other than `file not available` (if the file actually not available). Multiple read locks can be acquired on a file. An issue can arise if you are trying to open the file in append mode or write mode, in that scenario you will have to wait until the previous user releases the lock(assuming the file is already open under, write/append mode). — Umesh Kaushik, Jun 21 '18 at 06:31
I am trying to do: shutil.move(f, destination_dir) and this is what throws the exception. So basically, I want to move the file once I am sure the file is no longer used. — mothership, Jun 21 '18 at 06:37

Sunitha · Accepted Answer · 2018-06-21T08:25:38.420

3

You can use the retry module for these kind of retrying. This makes the code to look much cleaner. pip install retry should install the module

from retry import retry
import shutil

@retry((FileNotFoundError, IOError), delay=1, backoff=2, max_delay=10, tries=100)
def attempt_to_move_file(fname, dest_path):
    # Your attempt to move file
    # shutil.move(fname, destination_path)

With the above code, when attempt_to_move_file is invoked it would be retried (upto a max of 100 tries) whenever we hit FileNotFoundError or IOError and the retry happens with a sleep 1, 2, 4, 8, 10, 10, 10 ... seconds between attempts

edited Jun 21 '18 at 08:25

answered Jun 21 '18 at 07:00

Sunitha

11,777
2
20
23

shutil.move() will still move the file irrespective of the file is open or not, however, if any write operation is happening on the file, it will still keep running on the file in the new location. – Umesh Kaushik Jun 21 '18 at 07:15
no, the shutil.move() will not move the file if the file is used by other process. – mothership Jun 21 '18 at 07:37
@mothership please comment I made under my answer. – Umesh Kaushik Jun 21 '18 at 08:39
Just for the record, as @UmeshKaushik has pointed out using `shutil.move()` would **not** be the best solution as it can move partially created files.. Atleast thats the behavior I would see on Linux machines. As few others have pointed out, the best solution is to make the other process creating the file to create and acquire a lock on that file till it completes the writing. You should then wait till this lock getting released in your code – Sunitha Jun 21 '18 at 08:46
@Sunitha ... I can not influence the writter so I can not force it to set lock etc. hence why my approach to deal with it from the reader perspective. – mothership Jun 21 '18 at 09:10
@Umesh Kaushik ... please load the code to github/bitbucket etc. instead of google.drive as I can not read google.drive – mothership Jun 21 '18 at 09:10
@mothership it is a video and it is publically open https://drive.google.com/open?id=1zsWxVOGS5_lmcUW9CGA9Z96ZYLPlR1Le . and to resolve your issue, check this one https://stackoverflow.com/questions/11114492/check-if-a-file-is-not-open-not-used-by-other-process-in-python/11115521 , here you can filter out the if a process is using the file or not. – Umesh Kaushik Jun 21 '18 at 09:38

Umesh Kaushik · Answer 2 · 2018-06-21T07:17:57.913

0

you do not need to open a file to move it. Because once you have opened a file it is in open state hence you can not move it, Take it like this, when you are playing a video in a player and you try to delete it or cut and paste system won't allow you.

import shutil

file_name = 'trytry.csv'

shutil.copyfile(file_name, '/Users/umeshkaushik/PycharmProjects/newtry.csv')
shutil.move(file_name, '/Users/umeshkaushik/PycharmProjects/newtry1.csv')

Above code runs fine. Just make sure you are providing correct input and output path.

edited Jun 21 '18 at 07:17

answered Jun 21 '18 at 06:54

Umesh Kaushik

1,551
1
16
18

If process A creates the file and write lines to it then how do you check with above code if the process A still is not writting to the file so how do you check if the file is not opened or how do you check the file is not used and the writting is completed? The above code will take the snapshot of the file at the specific point in time while actually writting may be completed few seconds after you take the snapshot. – mothership Jun 21 '18 at 06:57
hence why I do try to open the file and then close it then I move it. – mothership Jun 21 '18 at 06:58
Your statements in your code are wrong. I am not creating the file. Other process creates the file. I am just opening it in read mode to see if I can open it. I close it straight after I open it and then I move it once I am sure I can open & close it so it tell me file is not used. – mothership Jun 21 '18 at 07:00
I agree, made a mistake by using write in the answer! Removing that part. – Umesh Kaushik Jun 21 '18 at 07:17
And for the records, multiple write locks are possible on a file. – Umesh Kaushik Jun 21 '18 at 07:18
what does it tell me? How to avoid multiple write locks if process1 (Scala) creates txt file and process1 (Python) picks the file only if the file writting is completed? – mothership Jun 21 '18 at 07:21
I am not sure about the scala, the scenario I mentioned is in python. 1. multiple write locks are possible and all will write in the file. 2. shutil.move will move the file irrespective of locks but will keep writing on the new location as well. However, if you would like to enforce the hard condition that file should not be moved if it is used by any process, I guess this might help you https://stackoverflow.com/questions/11114492/check-if-a-file-is-not-open-not-used-by-other-process-in-python/11115521 – Umesh Kaushik Jun 21 '18 at 07:31
why do you refer to multiple locks if I am not multithreading or multiprocessing with Python. I did not say it. I am running single process with Python trying to move the file if it is not used by other process which creates the file. The other process can be e.g. Scala, Java, Cobol or whatever. – mothership Jun 21 '18 at 07:39
shutil.move() will not move the file if the file is used by other process. – mothership Jun 21 '18 at 07:39
made this just to show you: let me know if you face any issue while viewing this: https://drive.google.com/file/d/1zsWxVOGS5_lmcUW9CGA9Z96ZYLPlR1Le/view?usp=sharing This is to show shutil will move the file. – Umesh Kaushik Jun 21 '18 at 07:52
and I was referring to multiple locks, to cover all the scenarios. – Umesh Kaushik Jun 21 '18 at 07:55

Python retry to open the file

2 Answers2