Will the file be fully read and written using below for loop in python?

Question

Trying to open a file to read and another to write into using below syntax.

Not clear if the file will be read completely and written completely before inserting record into the table (docs) or it will read only the buffered size 1024, write into the (dest) file and create the row in the table.

Want to read many files in a folder/ subfolders and create entry for each file in the table.

Or is it advisable to read the file in chunks and add the chunks together and write in (dest) file at once ?

If I don't specify the buffer size, reading will be limited by available RAM in the disk.

with open(os.path.join(db_path,filename),'rb') as src, \
                open(os.path.join(upload_folder,filename), 'wb') as dest:
           for chunk in iter(lambda: src.read(4096), b' '):
              dest.write(chunk) 
           if 1: # inserting record into a table
                ins = docs.insert().values(
                    file_name = filename,
                    up_date=datetime.datetime.utcnow())

I have modfied with what I could understand from your comment. Can you please let me know if it is right now?

can you please help us with information on which library is being used for 'docs' object ? — jazz, Oct 08 '18 at 06:08
@jazz I am using sqlalchemy core. docs is reference to the model table name — user956424, Oct 08 '18 at 06:11

Emil Vikström · Accepted Answer · 2018-10-08T08:23:43.700

2

You will read one chunk up to 1024 bytes. If you want to read the entire file you need to run read in a loop until it does not return anything.

Reading and writing one chunk at a time is advisable since your program can re-use the same memory and buffers instead of allocating more. You can experiment with the buffer size, though. I would not go below 4096, and the optimum may be somewhere around 8k to 16k. The optimum is determined by buffering in hardware and kernel buffer and page sizes.

edited Oct 08 '18 at 08:23

answered Oct 08 '18 at 06:26

Emil Vikström

90,431
16
141
175

https://stackoverflow.com/users/238978/emil-vikstr%c3%b6m I have modified the above code as per the link. Can you please check if it is correct? – user956424 Oct 08 '18 at 06:33
1

@user956424 You''ll figure it out. – Emil Vikström Oct 09 '18 at 08:39

Will the file be fully read and written using below for loop in python?

1 Answers1