Python - What is the fastest way i can copy files from a source folder to multiple destinations

Question

source = /tmp/src contains a,b,c,d files destinations = '/one' , '/two'

so i want to copy files a,b,c,d to both destinations '/one' and 'two'

something like

source = '/tmp/src'
destinations = []

def copy_files_multiple_dest(source,destinations)

right ? now, how would i loop through all the destinations

Will the files ever need to diverge? You might get aware with hardlinking if you're 100% sure they will always be the same after the copy. — Daenyth, Feb 01 '12 at 22:19
The source stays the same , since its populated by 'apt-get install' and whatever is in the repo folders, which is all packaged up in a .deb file — kamal, Feb 01 '12 at 23:14

score 1 · Accepted Answer · answered Feb 01 '12 at 22:15

1

how about something like:

import os
import shutil


source = '/tmp/src/'
destinations = []

def copy_files_multiple_dest(source,destinations):
  sfiles = os.listdir(source) # list of all files in source
  for f in sfiles:
    for dest in destinations:
      shutil.copy(os.path.join(source,f), dest)

i'm not sure it's the fastest but it should do the job.

answered Feb 01 '12 at 22:15

yurib

8,043
3
30
55

i get error[Errno 21] Is a directory: '/tmp/1/', when destinations = ['/tmp/1/','/tmp/2/'], code is appended to yours above – kamal Feb 03 '12 at 23:56
@kamal as the error message says, you're trying to move a directory. you can add a condition before the move function: if os.path.isfile(dest):.... – yurib Feb 04 '12 at 08:58
i had to use .strip() since argparse only supports strings and not lists. In my code, i was saving the space delimeted directory PATHs in a list – kamal Feb 07 '12 at 14:06

score 1 · Answer 2 · edited May 23 '17 at 12:03

Reading the source files only once makes sense here:

def xcopy_to_multiple_destinations(srcDir, destinations):
    for filename in os.listdir(srcDir):
        with open(os.path.join(srcDir, filename), "rb") as srcFile:
            for destDir in destinations:
                with open(os.path.join(destDir, filename), "wb") as destFile:
                    # ...copy bytes from srcFile to destFile...

If you want to copy recursively, use os.walk (see other question: Python recursive folder read). You can adapt the solution accordingly.

Note that "fastest" is a broad term. Hard linking should be faster, for example ;) Or using copy-on-write with the appropriate file system.

score 1 · Answer 3 · answered Feb 02 '12 at 01:10

os package is the usual way to got but have a look at this new project https://github.com/amoffat/pbs. you could then just do:

import pbs
destinations =['/one', '/two']
for destination in destinations:
   pbs.copy("-R", '/tmp/src', destination)

maybe not the fastest but certainly wins the beauty contest

score 0 · Answer 4 · answered Feb 02 '12 at 20:45

Since you say the files won't diverge, you can just hardlink them. You don't need python specifically, here's how I'd do it in bash.

dests=(a b c d)
for dest in "${dests[@]}"; do
  cp -rl /source/root "$dest"
done

If it has to be python, look at os.link and other functions in that module.

Python - What is the fastest way i can copy files from a source folder to multiple destinations

4 Answers4