How to copy 700 files + 80 files into two different directory from same source, using loop in Python?

Question

I want to copy, 700 files in folder Train and 80 files in folder Test from a source folder Z ( Z has 780 files). Can you please solve these using loops, Thanks in advance

I tried using this code:


sourceImage = "D:/DeveloperWorld/a-zDatasets"

train_folder = "D:/CheckFolder/train"
test_folder = "D:/CheckFolder/test"

src_files = os.listdir(sourceImage)

length = len(src_files)
for f in src_files:
    full_file_name = os.path.join(sourceImage, f)
    for i in range(0,700):
        shutil.copy(full_file_name, train_folder)
        break
    for i in range(700,780):    
        shutil.copy(full_file_name, test_folder)
        break

First, where are these files? Is what you are trying to achieve better described as your files are in folder A and you want to put them in folder B and C (which are in folder A) based on certain criteria? You mention that a for loop is slow. How big are these files? — Lewis Menelaws, Nov 12 '19 at 05:18
Can you please share what you have tried so far ? That will be useful to answer your questions — backtrack, Nov 12 '19 at 05:20

score 0 · Answer 1 · answered Nov 12 '19 at 05:24

0

This is shell command

cp -r path/to/A/* path/to/Z/

This means

copy recursively all contents of source A (notice *) to destination Z

You can use os to execute shell commands in your python code

import os 
os.system("cp -r path/to/A/* path/to/Z/")
os.system("cp -r path/to/B/* path/to/Z/")

answered Nov 12 '19 at 05:24

Yugandhar Chaudhari

3,831
3
24
40

Although your answer is correct, it really isn't a "python" solution. Essentially, writing a couple of shell commands and wrapping them in a python function makes the python aspect redundant. – Lewis Menelaws Nov 12 '19 at 05:38
I am doing deep learning algorithm so manually copying all the files would be time consuming , I want to automate it using loop. This seems shell command Can you help with other solution. – Sushan Bastola Nov 12 '19 at 05:47
@SushanBastola it is not manually copying replace with the paths on run time in your code itself. You are accomplishing the same no need to stick to using loops for this. – Yugandhar Chaudhari Nov 12 '19 at 05:57
There are other alogorithms which requires 1000+ of data, manually doing it will take whole lot time. Thanks anyway – Sushan Bastola Nov 12 '19 at 06:01
@SushanBastola I do know what training and testing algos need can you please make clear what you mean by *manual* ? So that I can help in explaining that this is already automated – Yugandhar Chaudhari Nov 12 '19 at 06:03
I mean manually copying and pasting 1000s of files from specific folder to train and test will take a lots of time. – Sushan Bastola Nov 12 '19 at 06:04
Yes so u have a script for this each time you call above code it will simply copy you dont need to do each time. Please try above code it does the same for you – Yugandhar Chaudhari Nov 12 '19 at 06:06
One more plus add this to cron and it will do that lets say per minute for you :) see https://stackoverflow.com/questions/8727935/execute-python-script-via-crontab – Yugandhar Chaudhari Nov 12 '19 at 06:08

score 0 · Accepted Answer · answered Nov 12 '19 at 06:11

It seems like what you are trying to do is move all these 780 files by using loops. The issue is when you are using the nested loops, you are looping perhaps hundreds of thousands of times because of this.

It seems like you are already getting a list of files from the src_files variable. Try this.

sourceImage = "D:/DeveloperWorld/a-zDatasets"

train_folder = "D:/CheckFolder/train"
test_folder = "D:/CheckFolder/test"

src_files = os.listdir(sourceImage)

length = len(src_files)
for idx, f in enumerate(src_files):
    full_file_name = os.path.join(sourceImage, f)

    if idx <= 700:
        shutil.copy(full_file_name, train_folder)
    else:
        shutil.copy(full_file_name, test_folder)

The enumerate function gives you an index.

How to copy 700 files + 80 files into two different directory from same source, using loop in Python?

2 Answers2