0

My target is to search the files containing a pattern and copy it to a folder. The files may available in many folders from a folder (say /home/abc/logs/ ....). Some file names and folders carry spacial characters including white spaces. I am trying to assign the result found to a variable and copy it. But result is "Argument list too long"

Linux server

pattern="somesttring" ;
LOG_FILE_PATH="/path/to/logs/"
LOGS_TO_COPY=$(grep -Hrn "$pattern" "$LOG_FILE_PATH" | cut -d: -f1)
cp $LOGS_TO_COPY temp #temp is folder to which i have to copy

I am getting the below error:

cp $LOGS_TO_COPY temp/
-bash: /bin/cp: Argument list too long
oguz ismail
  • 1
  • 16
  • 47
  • 69
impika
  • 107
  • 3
  • 11
  • @tripleee: I can't see either of those questions being a dupe. Quoting alone will not preserve the distinction between argument separator and bona-fide whitespace in the output of `grep`. If I am missing which part of the dupes answers this question's problem, please demonstrate. (Note that the immediately obvious answer of `cp "$LOGS_TO_COPY" temp/` is wrong.) – Amadan Sep 25 '19 at 09:43
  • Added another duplicate. You seem to be looking for `grep -rl "$pattern" "$LOG_FILE_PATH" | xargs cp -t temp/` – tripleee Sep 25 '19 at 09:46
  • @tripleee I'd be happy with any of the options I listed. That link is at least relevant, though not directly a solution of this (`grep` having `--null` option needs to be at least a comment). Given that producing the correct solution is clearly not as trivial as it seems, I'll vote to reopen, though the [link](https://stackoverflow.com/questions/11289551/argument-list-too-long-error-for-rm-cp-mv-commands) is quite useful to explain the whys and the generalities. (My other comment was deleted as incorrect.) – Amadan Sep 25 '19 at 09:54

1 Answers1

0

Don't store it in variable, at least not a normal string variable. There is no way for the string variable to keep the difference between the space inside each argument and the space delimiting them.

grep -lr "$pattern" "$LOG_FILE_PATH" | while IFS= read -r file; do cp "$file" temp/; done

Also note that you don't need to cut the grep result, as the -l flag gives you just the file names, exactly what you wanted. -H is the default so it's not needed, and -n is unnecessary if you're just going to cut it off.

This will not work if you have files containing newline in their names. As a solution, you can tell grep (and read) to use NUL for delimiter.

grep -lr --null "$pattern" "$LOG_FILE_PATH" | while IFS= read -d $'\0' -r file; do cp "$file" temp/; done

Alternately, you can use xargs instead of while+read:

grep -lr --null "$pattern" "$LOG_FILE_PATH" | xargs --null -I {} cp {} temp/

If you have to store it in a variable, use the same pattern as above to store it into an array variable. This way space is not needed as the argument delimiter.

declare -a files
while IFS= read -d $'\0' -r file; do files+=("$file"); done < <(grep -lr --null "$pattern" "$LOG_FILE_PATH")
cp "${files[@]}" temp/

Here's another way to do it (though this will not work if filenames include newline characters, as mapfile does not support changing the delimiter):

mapfile -t files < <(grep -lr "$pattern" "$LOG_FILE_PATH")
cp "${files[@]}" temp/

If the number of files to copy is very big, the while and xargs solutions will still work, but those that simply plop the entire array into one cp command may not.

Amadan
  • 191,408
  • 23
  • 240
  • 301
  • Thanks for the solution. Here, when i run the command --> grep -Hrn "$pattern" "$LOG_FILE_PATH" number of files found is 22584(in my case) where when i ran your command(1st command with while), the number of files found is 329 SO i feel many files are missing – impika Sep 25 '19 at 09:37
  • As I said, `grep -Hrn` shows each line where there is a match. You then cut off the line and the content using `cut`, so if 100 lines match, you will have the same file name 100 times. I haven't thought about it, but this will also be the reason your command line is too long. `grep -l` will only display each file name once. Try to compare `grep -lr ... | wc -l` with the deduped version of your list: `grep -Hrn ... | cut ... | sort | uniq | wc -l`. – Amadan Sep 25 '19 at 09:39
  • Pasting here the results obtained grep -lr "$pattern" "$LOG_FILE_PATH" | while IFS= read -r file; do cp "$file" .; done ll | wc -l 330 ############## grep -Hrn "$pattern" "$LOG_FILE_PATH"| sort | uniq | wc -l 22598 ############### grep -Hrn "$pattern" "$LOG_FILE_PATH" | wc -l 22598 – impika Sep 25 '19 at 10:54
  • You missed the cut sort uniq step to see how many _different_ files you’re getting. – Amadan Sep 25 '19 at 10:54