Why is xargs not replacing the second {}

Question

I'm using xargs to try to echo the name of a file, followed by its contents. Here is the command

find output/ -type f | xargs -I {} sh -c "echo {} ; cat {}"

However for some reason, the second replace after cat is not being replaced. Only for some files, so some files do work correctly.

To be clear, I'm not looking for a command that lets me echo the name of a file followed by its contents, I'm trying to understand why this specific command does not work.

https://stackoverflow.com/questions/36224777/xargs-command-length-limits/45595216#45595216 for linking — Carlos Bribiescas, Aug 09 '17 at 15:43

score 17 · Accepted Answer · edited Oct 07 '20 at 23:56

17

Turns out that the command was too long, so it was working with shorter file names and failing for longer ones. From man xargs

-I replstr
             Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility with the
             entire line of input.  The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concatenating as much
             of the argument containing replstr as possible, to the constructed arguments to utility, up to 255 bytes.  The 255 byte limit does not apply to arguments to utility
             which do not contain replstr, and furthermore, no replacement will be done on utility itself.  Implies -x.

edited Oct 07 '20 at 23:56

ffledgling

11,502
8
47
69

answered Aug 09 '17 at 15:39

Carlos Bribiescas

4,197
9
35
66

7

Also from the manpages: -S replsize Specify the amount of space (in bytes) that -I can use for replacements. The default for replsize is 255. I was able to use xargs with a long command by specifying `xargs -S 100000 -I {} `. – Jacob Rodal Jul 28 '21 at 03:21
2

I dug into this more and learned that I didn't run into a command length issue when running on Linux, only mac os. The manpages I mentioned above are from mac os. – Jacob Rodal Jul 28 '21 at 03:35

score 4 · Answer 2 · answered Feb 28 '19 at 06:47

The root cause of the problem is pointed out in Carlos' answer, but without a solution.

After some googling, I couldn't find a way to lift up the 255 characters limit.

So a probable way to workaround it, is to use shell variable as a substitution.

Example:

find . | xargs -I% sh -c 'F="%";iconv -f gb2312 -t utf-8 "$F">"$F.out";mv "$F.out" "$F"'

Remember to use single quotes at the outermost sh -c parameter string, we don't want the $F inside to be replaced by our parent shell.

you can escape the $ in front of a variable to make the subshell fill it instead of the parent shell if you have to use double quotes — Eric Jones, Oct 23 '19 at 19:49

score 2 · Answer 3 · answered Aug 09 '17 at 15:20

2

Is it files with white space in the name that create problems? Try adding \", like this:

find output/ -type f | xargs -I {} sh -c "echo \"{}\" ; cat \"{}\""

This worked for me using Bash.

answered Aug 09 '17 at 15:20

Mats

101
5

Good idea, but there are not spaces in my case – Carlos Bribiescas Aug 09 '17 at 15:21
This should be a comment. You should first ask for details if needed. Please read this: [answer] – tupui Aug 09 '17 at 15:29
Assuming you're using Bash, if you use `set -x` and then run your command, what is the output for a file name that does not work? – Mats Aug 09 '17 at 15:29

Why is xargs not replacing the second {}

3 Answers3