Read file from folder which is being written python

Question

I have a folder which is being written continuously with files (from a Raspberry Pi). I want to run a Python code which will read files (and process them) from the same folder. I start the Python code after say 5 files have been written into the folder. But then the code stops after it has processed 5 files. Any suggestions as to how to "stream" the files from this dynamically updated folder into my Python code?

Thanks

I am using the following code and was wondering if there is another solution:

import os
import time
import numpy as np
from PIL import Image
import datetime

# get the data & time of an image output is str with format '%Y:%m:%d 
%H:%M:%S'
def get_date_taken(path):
    return Image.open(path)._getexif()[36867]

# get the difference in time between two images in seconds
def getTimeDiffBetwImages(path1,path2):
    a1 = get_date_taken(path1)
    a2 = get_date_taken(path2)
    b1 = datetime.datetime.strptime(a1,'%Y:%m:%d %H:%M:%S')
    b2 = datetime.datetime.strptime(a2,'%Y:%m:%d %H:%M:%S')
    return (b1-b2).total_seconds()

def getDiffImgs(img1,img2):
    a = img1.shape
    diff = np.zeros((a[0],a[1],a[2]),dtype = "uint8")
    for i in range(0,a[0]):
        for j in range(0,a[1]):
            for k in range(0,a[2]):        
                diff[i,j,k] = abs(int(img2[i,j,k]) - int(img1[i,j,k]))
    return diff

folder = "D:/work"
file_num = 0
img_prev = []
last_file = None
while True:
    for f in os.listdir(folder):
        path1 = os.path.join(folder,f) 
        if not(last_file == None):
            path2 = os.path.join(folder,last_file) 
            time_diff = getTimeDiffBetwImages(path1,path2)
        else:
            time_diff = 0
        if  time_diff >= 0:
            print(f)    
            im = cv2.imread(os.path.join(folder,f))
            img_curr = np.copy(im)
            fname = "Diff_" + f
            if file_num > 0:
                diff = getDiffImgs(img_prev,img_curr)
                cv2.imwrite(fname,diff)
            img_prev = np.copy(img_curr)
            file_num += 1
            time.sleep(10) # additional processing but just delay out here
            last_file = f

This question is quite broad. Can you please provide some code so we can see what you're working with. Also please provide context as to what you have, what you've tried, and what you'd like to see. — , Oct 03 '18 at 22:34
Welcome to StackOverflow. Please read and follow the posting guidelines in the help documentation, as suggested when you created this account. [On topic](http://stackoverflow.com/help/on-topic), [how to ask](http://stackoverflow.com/help/how-to-ask), and [... the perfect question](https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/) apply here. — Prune, Oct 03 '18 at 22:44
Thanks for all the suggestions. I am a new user and will try best to adhere to the rules of the forum! — Shrini, Oct 03 '18 at 22:56

score 0 · Answer 1 · answered Oct 03 '18 at 22:42

0

If you are reading the folder when the 5 files have been added. It may keep a log of those five files only. Any new files added will not be reflected. I think a good solution would be to main a set() of files being added to directory. While processing the files from the set, if the last file comes, you can check the directory again for any new files and add them to that set. Keeping this in a loop until your preferred condition is not met.

I hope it helps. Thanks.

answered Oct 03 '18 at 22:42

Rushin Naik

36
1

Thanks a lot for quick response. I was actually using a small function to get the time the image was taken and using the time difference (last time the folder was opened and storing the last file) to check if it is >= 0 to proceed with the file or not. I haven't checked if it works :-), but was hoping if there was another solution. – Shrini Oct 03 '18 at 22:55

Read file from folder which is being written python

1 Answers1