How to bash copy all files in directory/sub-directories excluding those contained in a list while flattening the tree structure?

Question

I have a directory (called 'source') that contains sub-directories and files. Using bash I need to copy all files (and only files, not directories) found in this directory and each of its sub-directories to a different directory (called 'destination'). The directory tree must not be maintained/must be flattened. Only files that are not included in a text file (called 'excluded.txt') must be copied.

Source input examples:

/home/source/AAA/file1.xyz 
/home/source/AAA/GGG/file2.xyz
/home/source/BBB/file3.tuv
/home/source/BBB/HHH/file4.tuv

Destination output examples:

/home/destination/file1.xyz
/home/destination/file2.xyz
/home/destination/file3.tuv
/home/destination/file4.tuv

Once the files have been copied, the four+ filenames (file1.xyz, etc,) are added to excluded.txt (with each filename on a new line). The files will then be removed from destination directory periodically.

If the bash script is executed again, and source files are present, they should not be copied to destination if their filenames appear in the excluded.txt file.

I have failed by attempting to us "cp" and "rsync", as the directory tree structure was maintained. I have also failed using "find", as I haven't been able to check the results against the "excluded.txt" list before taking the copy action.

find /home/source/ -type f -exec cp -n {} /home/destination/ \; If you want to avoid overwrite of existing files in destination folder, use 'cp -n' — Kuldeep Singh, Mar 14 '17 at 09:43
@KuldeepSingh I initially wrote this code snippet as an answer, but I forgot to take into account that OP had a file with excluded files — Aserre, Mar 14 '17 at 09:44
One can avoid rewriting of files with -n option of cp command. Unless excluded.txt is required somewhere else — Kuldeep Singh, Mar 14 '17 at 09:47
@KuldeepSingh Thank you for your proposed solution! It would have worked if the files remained in the destination directory. However, because the files are periodically removed, the excluded.txt list keeps track of all moved/copied files. — Bob, Mar 14 '17 at 21:58

Aserre · Answer 1 · 2017-03-15T09:18:34.470

1

find should be the tool to use for recursive search

find /home/source -type f $(printf "! -name %s" "$(cat exclude.txt)") -exec cp -n {} /home/destination \;

Explanation :

find /home/source : the path of the root directory to search. Search is recursive.
-type f : only retrieve files
$(printf "! -name %s " $(cat exclude.txt)) : will write ! -name file1 ! -name file2 ..., listing all the files to exclude
-exec cp -n {} /home/destination : action executed for each found item. {} represents the item that was found.

edited Mar 15 '17 at 09:18

answered Mar 14 '17 at 09:37

Aserre

4,916
5
33
56

Perfect @Aserre – Kuldeep Singh Mar 14 '17 at 11:39
@Aserre This works perfectly. Having the excluded list prevents the files from being copied repeatedly (since they are removed periodically from the destination location). Thank you very much! – Bob Mar 14 '17 at 21:55
@Aserre So, I just determined that this code works only for filenames that do not contain spaces. If there are spaces in the filename, like "file 2.txt", the file is repeatedly copied to the destination directory (iff the original copy has been removed from the destination directory) when code is executed. – Bob Mar 14 '17 at 23:27
@Aserre Thanks for your help so far. Unfortunately this doesn't take care of the issue. If I have two files in the source directory, "file1.txt" and "file 2.txt". The new output from the `$(printf "! -name %s" "$(cat exclude.txt)"` command shows `! -name file1.txt` followed by `'file 2.txt'` on the next line. `'file 2.txt'` should be on the same line with `! -name ` preceding it. I've been searching the forums for a solution and it seems more difficult than I anticipated. Any thoughts? – Bob Mar 16 '17 at 20:43

score 0 · Accepted Answer · edited May 23 '17 at 12:25

The answer provided by @Aserre was instrumental in finding this solution. His solution will work for all files that do not contain spaces. After reading about eval (evaluating/executing strings), string concatenation, and how to read entire lines into variables, I was able to write and execute the following code successfully.

while read -r line
do
    name="$line"
    exclude="$exclude ! -name \"$name\""
done < "/mnt/destination/exclude.txt"
cmd1="find \"/home/source\" -type f "
cmd2=" -exec cp -n {} \"/home/destination\" \;"
result=$cmd1$exclude$cmd2
eval $result

Explanation (credit to @Aserre):

while read -r line : go through every line in exclude.txt. "-r" flag causes the backslash to be considered to be part of the line.
name="$line" : the entire line in excluded.txt is stored in a new string called "name".
exclude="$exclude ! -name \"$name\"" : will store ! -name "file1" ! -name "file2" ! -name... in a new string called "exclude". This string is a list of all the files to exclude, each preceded by ! -name. The backslash is necessary before each of those quotation marks.
cmd1= : store the following 2 commands into a string called "cmd1".
find /home/source : the path of the root directory to search. Search is recursive.
-type f : retrieve files only.
-exec cp -n {} /home/destination : action executed for each found item. {} represents the item that was found.
cmd2= : stores the previous command into a string called "cmd2".
result=$cmd1$exclude$cmd2 : concatenates all 3 strings.
eval $result : take the string named "result" and run it as a command.

How to bash copy all files in directory/sub-directories excluding those contained in a list while flattening the tree structure?

2 Answers2