Modifying replace string in xargs

Question

When I am using xargs sometimes I do not need to explicitly use the replacing string:

find . -name "*.txt" | xargs rm -rf

In other cases, I want to specify the replacing string in order to do things like:

find . -name "*.txt" | xargs -I '{}' mv '{}' /foo/'{}'.bar

The previous command would move all the text files under the current directory into /foo and it will append the extension bar to all the files.

If instead of appending some text to the replace string, I wanted to modify that string such that I could insert some text between the name and extension of the files, how could I do that? For instance, let's say I want to do the same as in the previous example, but the files should be renamed/moved from <name>.txt to /foo/<name>.bar.txt (instead of /foo/<name>.txt.bar).

UPDATE: I manage to find a solution:

find . -name "*.txt" | xargs -I{} \
    sh -c 'base=$(basename $1) ; name=${base%.*} ; ext=${base##*.} ; \
           mv "$1" "foo/${name}.bar.${ext}"' -- {}

But I wonder if there is a shorter/better solution.

No, except that I would use more quoting `mv "$1" "foo/${name}.bar.${ext}"` and you could do `basename` like this: `base=${1##*/}`. You should post your solution as an answer and accept it. — Dennis Williamson, May 29 '12 at 17:58
@DennisWilliamson Thanks for your comment! I will wait a bit more, just to see if anyone comes up with some fancy thing, otherwise I will answer the question myself. — betabandido, May 29 '12 at 18:34
I think if your filename is the last thing on the line, you don't need the -I{} nor {} in the command line. (Note that the very purpose of xargs is to group though, so if you DON'T want multiple things at the end of 1 xargs' argument invocation you need "-l 1" (or -L 1 for some versions of xargs). -I{} implies -l 1, so that's why that also works here.) — Michael Campbell, May 29 '12 at 19:43
The `sh -c` was the key for me! Thanks! Else my cut/sed command in `echo {}|sed '...'` was not executed — Julian, Nov 03 '18 at 16:19

score 56 · Answer 1 · answered Feb 21 '14 at 08:10

56

The following command constructs the move command with xargs, replaces the second occurrence of '.' with '.bar.', then executes the commands with bash, working on mac OSX.

ls *.txt | xargs -I {} echo mv {} foo/{} | sed 's/\./.bar./2' | bash

answered Feb 21 '14 at 08:10

fairidox

3,358
2
28
29

I'd use `echo` instead of `ls` here. – Tripp Kinetics Aug 02 '20 at 21:17
@TrippKinetics can you please elaborate on how echo would be used here? And why? – Asrail Oct 11 '22 at 18:09
`ls` output is not meant to be processed directly by text processing tools. It is designed specifically for user output. It's behavior when output to text processing tools isn't always consistent. Instead of `ls *.txt | pipecommand`, it is preferable to `echo *.txt | pipecommand`, or even sometimes `find . -type f -name '*.txt' -print | pipecommand`. – Tripp Kinetics Oct 21 '22 at 16:35

score 35 · Answer 2 · answered Mar 09 '16 at 21:40

35

It is possible to do this in one pass (tested in GNU) avoiding the use of the temporary variable assignments

find . -name "*.txt" | xargs -I{} sh -c 'mv "$1" "foo/$(basename ${1%.*}).new.${1##*.}"' -- {}

answered Mar 09 '16 at 21:40

Santrix

875
8
10

5

I love the way you are using the replace string as a positional parameter to the shell so that you can perform substitutions and expansions. Very cool. – GL2014 Sep 02 '16 at 14:42
Curious to know why did you use -- {} instead of simple {} in the end. I mean why not ? find . -name "*.txt" | xargs -I{} sh -c 'mv "$0" "foo/$(basename ${0%.*}).new.${1##*.}"' {} – mac Sep 12 '20 at 15:14
1

@mac: (necroposting, sorry !) .. because when processing files which fit the pattern `*.txt`, you can never know in advance what's coming at you. You may get files that start with "-" and you want to avoid confusing them with options. – Cbhihe Apr 13 '21 at 09:22
can someone explain how the `-- {}` at the very end gets applied? is it operating on `xargs` or `sh`? – Collin Mar 03 '22 at 08:38
@CollinGonzalez the `--` tells the `sh` command that no further options are coming, just in case one of the files starts with `-`. The final `{}` is substituted for the argument that xargs receives via the `-I{}` option. This in turn is passed to the subprocess which runs the `mv` command as $1. – Santrix Mar 04 '22 at 09:14
1

@Santrix I think there is an important thing in your answer in this particular case. when invoking `sh` with the `-c` command the first argument will be given to `$0`. so if you didn't have `--` there then your actual file path argument would be put in `$0` instead of `$1`. – masonCherry Feb 22 '23 at 10:40

score 21 · Accepted Answer · answered May 29 '12 at 19:48

21

In cases like this, a while loop would be more readable:

find . -name "*.txt" | while IFS= read -r pathname; do
    base=$(basename "$pathname"); name=${base%.*}; ext=${base##*.}
    mv "$pathname" "foo/${name}.bar.${ext}"
done

Note that you may find files with the same name in different subdirectories. Are you OK with duplicates being over-written by mv?

answered May 29 '12 at 19:48

glenn jackman

238,783
38
220
352

Yeah, I know it has that "problem". Actually the example that I posted is rather a silly one. I just made it up in order to show a simple example in my question, so no problem :) – betabandido May 29 '12 at 20:01
2

To handle filenames that has a newline in it, use this: `find . -name "*.txt" -print0 | while read -r -d '' pathname; do ...` – Cem May 18 '15 at 21:42
4

Note that this (serial) approach won't work if you are trying to parallellize xargs using `--max-procs`. – Nov 26 '15 at 12:03

Ole Tange · Answer 4 · 2022-02-22T10:31:31.063

12

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

find . -name "*.txt" | parallel 'ext={/} ; mv -- {} foo/{/.}.bar."${ext##*.}"'

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

edited Feb 22 '22 at 10:31

answered May 30 '12 at 19:48

Ole Tange

31,768
5
86
104

2

I also recommend GNU Parallel over `xargs`, it's much more powerful and flexible. As for its installation, I strongly suggest to use your distribution's package manager (such as `sudo apt-get install parallel`) or parallel's recommended install procedure (as per http://git.savannah.gnu.org/cgit/parallel.git/tree/README): `(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash` – MestreLion Dec 31 '15 at 11:56
6

@MestreLion Please do not recommend the curl-pipe-to-shell anti-pattern. What is `pi.dk`? Why should I trust that domain? `gnu.org` is at least more trustworthy. But neither URL is using TLS, and more importantly, neither one verifies a cryptographic signature of what's downloaded. The only solution to recommend is `apt-get`. – Mar 15 '16 at 04:27
As you can see pi.dk/3 does a verification of cryptographic signature and stops if the signature does not match. If you feel uncomfortable piping it directly to a shell you can save it to a file, inspect the code, and then run it. – Ole Tange Mar 16 '16 at 14:44
2

@blujay: this is not *my* recommendation, this was taken from ***the official documentation***: http://www.gnu.org/software/parallel/parallel_tutorial.html#Prerequisites . And while I agree that this is not as good as using your distro's package manager, it is *far better* than a simple `wget ... && chmod +x` – MestreLion Mar 16 '16 at 16:54
3

@MestreLion It's sad that it's on the Parallel site, and you shouldn't spread it further. It is not better than `wget ...` because it is [dangerous for many reasons](https://www.seancassidy.me/dont-pipe-to-your-shell.html). It's tantamount to telling a Windows user to run an `exe` from a random web site without scanning it for viruses. Just say no. – Mar 17 '16 at 15:49
@blujay: I agree the install script should be made available at the GNU website instead of `pi.dk`, if only for HTTPS access. Official instructions could then suggest `wget && chmod` on the *install* script (instead of directly on the parallel script) to avoid the pipe-to-shell. But that's a task for Ole Tange, parallel's maintainer :) – MestreLion Mar 22 '16 at 15:06

score 2 · Answer 5 · answered May 29 '12 at 18:31

If you're allowed to use something other than bash/sh, AND this is just for a fancy "mv"... you might try the venerable "rename.pl" script. I use it on Linux and cygwin on windows all the time.

http://people.sc.fsu.edu/~jburkardt/pl_src/rename/rename.html

rename.pl 's/^(.*?)\.(.*)$/\1-new_stuff_here.\2/' list_of_files_or_glob

You can also use a "-p" parameter to rename.pl to have it tell you what it WOULD HAVE DONE, without actually doing it.

I just tried the following in my c:/bin (cygwin/windows environment). I used the "-p" so it spit out what it would have done. This example just splits the base and extension, and adds a string in between them.

perl c:/bin/rename.pl -p 's/^(.*?)\.(.*)$/\1-new_stuff_here.\2/' *.bat

rename "here.bat" => "here-new_stuff_here.bat"
rename "htmldecode.bat" => "htmldecode-new_stuff_here.bat"
rename "htmlencode.bat" => "htmlencode-new_stuff_here.bat"
rename "sdiff.bat" => "sdiff-new_stuff_here.bat"
rename "widvars.bat" => "widvars-new_stuff_here.bat"

Thanks for your answer! `mv` example was just an example, though. I face the need to do something similar in quite many other situations, so I guess `rename.pl` would not work for those cases. — betabandido, May 29 '12 at 18:38
Well, yes and no; you can easily modify rename.pl to *NOT* actually rename a file, but do something else with it. As you can see, with the "-p" parameter, it doesn't rename at all, but just prints the result of the transformation. You can use that script as a basis to do many other things, and the argument that it takes is arbitrary perl code, which is immensely powerful. But you are certainly welcome nonetheless. — Michael Campbell, May 29 '12 at 19:39

score 2 · Answer 6 · answered May 01 '18 at 12:33

the files should be renamed/moved from <name>.txt to /foo/<name>.bar.txt

You can use rename utility, e.g.:

rename s/\.txt$/\.txt\.bar/g *.txt

^{Hint: The subsitution syntax is similar to sed or vim.}

Then move the files to some target directory by using mv:

mkdir /some/path
mv *.bar /some/path

To do rename files into subdirectories based on some part of their name, check for:

-p/--mkpath/--make-dirs Create any non-existent directories in the target path.

Testing:

$ touch {1..5}.txt
$ rename --dry-run "s/.txt$/.txt.bar/g" *.txt
'1.txt' would be renamed to '1.txt.bar'
'2.txt' would be renamed to '2.txt.bar'
'3.txt' would be renamed to '3.txt.bar'
'4.txt' would be renamed to '4.txt.bar'
'5.txt' would be renamed to '5.txt.bar'

score 2 · Answer 7 · answered Jul 23 '18 at 16:50

Adding on that the wikipedia article is surprisingly informative

for example:

Shell trick

Another way to achieve a similar effect is to use a shell as the launched command, and deal with the complexity in that shell, for example:

$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 bash -c 'for filename; do cp -a "$filename" ~/backups; done' bash

score 0 · Answer 8 · answered Jun 14 '16 at 08:54

0

Inspired by an answer by @justaname above, this command which incorporates Perl one-liner will do it:

find ./ -name \*.txt | perl -p -e 's/^(.*\/(.*)\.txt)$/mv $1 .\/foo\/$2.bar.txt/' | bash

answered Jun 14 '16 at 08:54

Vasiliy

16,221
11
71
127

Modifying replace string in xargs

8 Answers8

Shell trick

Linked