linux wildcard usage in cp and mv

Question

I am composing a script to process 20 files. All of them located in different directories. I have partial file name.

In log directory, File1_Date_time.err change to File1__Date_time_orig.err
cd ../scripts/
sh File.sh

File1 directory is /data/data1directory/Sample_File1/logs/File1_Data_time.err
File2 directory is /data/data2directory/Sample_File2/logs/File2_Data_time.err
.....

My script looks like this. (runrunrun.sh)

#!/bin/bash
INPUT=$1
mv /data/*/Sample_*/logs/*_Data_time.err /data/*/Sample_*/logs/*_Data_time_orig.err
cp /data/*/Sample_*/scripts/*.sh /data/*/Sample_*/scripts/*_orig.sh
sh /data/*/Sample_*/scripts/*_orig.sh

When running it, I tried.
./runrunrun.sh File1
. runrunrun.sh File1
sh runrunrun.sh File1

mv: cannot move /data/data1directory/Sample_File1/logs/File1_Data_time.err /data/*/Sample_*/logs/*_Data_time_orig.err: No such file or directory cp also got similar feedback

Am I doing it correct?

Thanks!

`mv`, `cp`, etc. **aren't passed the wildcards you're writing**, if those wildcards have any valid expansions. Instead, they're passed literal filenames created by running those expansions. Thus, since they don't know the literal wildcards used in the input name, they can't possibly map a `*` in the input name to a `*` in the output name. — Charles Duffy, Feb 29 '16 at 20:01
If you want to write renaming tools, a good place to start is BashFAQ #30: http://mywiki.wooledge.org/BashFAQ/030 — Charles Duffy, Feb 29 '16 at 20:02
Related: http://stackoverflow.com/questions/20629302/better-way-to-rename-files-based-on-multiple-patterns/25597051 — Charles Duffy, Feb 29 '16 at 20:03
BTW, `sh foo.sh` is **very dangerous**, unless you're absolutely certain that that script starts with `#!/bin/sh` rather than `#!/bin/bash`, `#!/bin/ksh`, `#!/bin/zsh`, etc; otherwise, it may use the wrong interpreter. — Charles Duffy, Feb 29 '16 at 20:08

Charles Duffy · Accepted Answer · 2017-12-23T20:54:13.083

Let's talk about how wildcards work for a minute.

cp *.txt foo

doesn't actually invoke cp with an argument *.txt, if any files matching that glob exist. Instead, it runs something like this:

cp a.txt b.txt c.txt foo

Similarly, something like

mv *.txt *.old

...can't possibly know what to do, because when it's invoked, what it sees is:

mv a.txt b.txt c.txt *.old

or, worse, if you already have a file named z.old, it'll see:

mv a.txt b.txt c.txt z.old

Thus, you need to use different tools. Consider:

# REPLACES: mv /data/*/Sample_*/logs/*_Data_time.err /data/*/Sample_*/logs/*_Data_time_orig.err
for f in /data/*/Sample_*/logs/*_Data_time.err; do
  mv "$f" "${f%_Data_time.err}_Data_time_orig.err"
done

# REPLACES: cp /data/*/Sample_*/scripts/*.sh /data/*/Sample_*/scripts/*_orig.sh
for f in /data/*/Sample_*/scripts/*.sh; do
  cp "$f" "${f%.sh}_orig.sh"
done

# REPLACES: sh /data/*/Sample_*/scripts/*_orig.sh
for f in /data/*/Sample_*/scripts/*_orig.sh; do
  if [[ -e "$f" ]]; then
    # honor the script's shebang and let it choose an interpreter to use
    "$f"
  else
    # script is not executable, assume POSIX sh (not bash, ksh, etc)
    sh "$f"
  fi
done

This uses a parameter expansion to strip off the tail end of the old name before adding the new name.

This is not the case anymore as of 2020 with GNU bash 5.0 and above — user164863, Dec 20 '20 at 17:48
@user164863, please provide a contrary demonstration. `mv` and `cp` are not part of bash itself, they're provided by your OS vendor, and globs are already expanded before they're invoked; none of that has changed. — Charles Duffy, Dec 20 '20 at 20:30

MikeW · Answer 2 · 2020-12-09T09:40:50.490

49

The find command can be used quite concisely in simple cases where you want to perform operations on wildcard (or more complex) filename matches. The technique below can be committed to memory ... almost !

This works by letting the find command run another command on each filename it finds. You can dry-run this example using echo instead of/in front of mv .

If we wanted to move all files in the current directory with name beginning 'report', to another parallel directory called 'reports' :

find . -name "report*.*" -exec mv '{}' ../reports/ \;

The wildcard string must be in quotes, the {} marking the filename that was 'found' must be in quotes, and the final semicolon must be escaped - all due to Bash/shell treatment of those characters.

Look at the man page for find for more uses: https://linux.die.net/man/1/find

edited Dec 09 '20 at 09:40

answered Feb 11 '19 at 16:41

MikeW

5,504
1
34
29

if the file names of multiple files are the same, this solution seems to overwrite the files in the destination. How can the entire folder structure be copied over? – Arthur Yip Feb 04 '22 at 20:21
1st answer covers my case. I also used cp --parents ============================ for f in ./*/*20220202.csv; do cp -rfv --parents "$f" "/projects/drprojects/evmc/tempo4/" done – Arthur Yip Feb 05 '22 at 00:00
The code was just an example. The original question also had that problem. Ask a separate question, or work it out ! – MikeW Feb 08 '22 at 15:54

Eduard Florinescu · Answer 3 · 2020-02-18T07:58:30.490

7

This is what I use, very fast to write and remember

For copying:

ls -1 *.txt | xargs -L1 -I{} cp {} {}.old

For moving:

ls -1 *.txt | xargs -L1 -I{} mv {} {}.old

the explanation:

xargs(1) -L1 -I{}

build and execute command lines from standard input

-L max-lines
       Use  at most max-lines nonblank input lines per command line.  Trailing blanks cause an input line
       to be logically continued on the next input line.  Implies -x.

-I replace-str
       Replace occurrences of replace-str in the initial-arguments with names read from  standard  input.
       Also,  unquoted  blanks  do  not  terminate  input  items;  instead  the  separator is the newline
       character.  Implies -x and -L 1.

Basically with -L1 it sends the arguments on by one, and -I{} makes {} the placeholder

edited Feb 18 '20 at 07:58

answered Dec 09 '19 at 12:14

Eduard Florinescu

16,747
28
113
179

2

Buggy with interesting filenames. In general, xargs is only safe with NUL-delimited input and the `-0` argument; default behavior splits input on spaces, not just newlines (though newlines aren't safe either, as they're allowed in filenames); treats quote characters as syntax rather than literal data; and otherwise generally munges input. – Charles Duffy Feb 18 '20 at 04:19
Also, see [Why you shouldn't parse the output of `ls`](https://mywiki.wooledge.org/ParsingLs) – Charles Duffy Feb 18 '20 at 04:23

score 3 · Answer 4 · answered Dec 23 '20 at 10:50

using awk and sed

Filter using ls, use awk to build your move command for each record and the sed 'e' in the end will execute the command.

ls  *.avro  | awk ' { print "mv "$1" aa_"$1 }' | sed 'e'

First try without the sed -- once you know your command is showing correctly add the sed.

score 0 · Answer 5 · edited Dec 23 '17 at 20:43

0

Try this:

Change these:

zc_cd_delim_01.csv
zc_cd_delim_04.csv

into these:

zc_cd_delim_113_01.csv
zc_cd_delim_113_04.csv

command line:

for f in zc_cd_delim_??.csv; do var1=${f%%??.*}; 
    mv $f "${var1}113_${f#$var1}"; 
done

edited Dec 23 '17 at 20:43

Dmitry

6,716
14
37
39

answered Dec 23 '17 at 19:59

Jon Lancelle

1
1

`mv "$f"`, rather -- if the `??` matches a filename with whitespace, or `IFS` contains a character found in the names (ie. `_`), behavior could otherwise be surprising. – Charles Duffy Dec 23 '17 at 20:50
BTW, I'm a little unclear on how this is applicable to the question at hand -- `c_cd_delim_??.csv` doesn't look to me like one of the patterns the OP was using as an example of what they wanted to match. Would you mind editing to make applicability to the question more clear? – Charles Duffy Dec 23 '17 at 20:52

linux wildcard usage in cp and mv

5 Answers5

Linked