Why do I get "usr/bin/cp(&find): Argument list too long"?

Question

Error: Argument list too long

sudo cp and find... -exec sudo cp yield

/usr/bin/sudo(or find): Argument list too long

I get this error in my Travis CI build output. My .travis.yml file uses a shell script (deploy.sh) to run a git diff against branch2's force-app folder, and the output (several thousand files) is cp into a new directory:

sudo cp --parents $(git diff --name-only branch2 force-app/) $DEPLOYDIRECTORY;

The error is two lines:

./deploy.sh: line 122: /usr/bin/git: Argument list too long
./deploy.sh: line 122: /usr/bin/sudo: Argument list too long

I have the same issue with the subsequent find command. I use a naming convention to find and copy a file that corresponds with each file from the git diff:

for FILE in $GIT_DIFF_FILES; do
    if [[ $FILE == *Test.cls ]]; then
        find $classPath -maxdepth1 -samefile "$FILE-meta.xml" -exec sudo cp --parents {} $DEPLOY_DIRECTORY +
    fi;
done;

Again, I get the error: ./deploy.sh: line 142: /usr/bin/find: Argument list too long.

Note: Travis CI is running an Ubuntu Linux (Xenial) virtual environment for this build.

---

What I've Tried

1. Resetting the stack size

Travis CI automatically sets the stack size to 8192 and the arg max to 8388608 for each new build. In my deploy.sh file, I ulimit -s 9999999 to change the stack size to 9999999 and the arg max to 2559999744 for each new build.

2. Command Variations

For the first sudo cp command, I've tried:

1. formatting command as a for loop
2. tar -cf - -C files... | tar xpf - -C target directory...
3. formatting command as git diff ... | xargs cp ...
   

For the find command, I've tried:

1. find ... | xargs -n 1000 sudo cp ....
   • Any different flags added to this command don't work either.
2. find ... -exec cp ...
   • Any flags or syntax changes return the same error

Changing the command doesn't appear to be the solution

After reproducing the error locally, I found that my commands already work fine when interacting with a smaller number of files (~1000 or less).

However, running the cp and find commands when the output of git diff is approximately 1200-1500 files or more then returns:

Argument list too long

In addition, simply running find or cp from my shell script also returns:

Argument list too long

It does not seem to matter what I do to these commands, therefore. Something more fundamental is causing the problem, but only when the file count of the git diff output exceeds approximately 1200 files.

How do I fix my cp and find commands?

---

How to Reproduce the Error

Here are the steps to take to reproduce this error according to my use case. I used VS Code, so I recommend you do as well to replicate this as similarly to me as possible. Note that you will need to replace USERNAME in the below code with your username.

Step 1: Fork this repo.

Step 2: Follow the steps below in your terminal

cd force-app/main/default
mkdir diff
git checkout -b branch2
cd force-app/main/default/classes

Add //comment to the bottom of myclass.cls and myclass.cls-meta.xml files. Save changes.

for n in {001..1500}; do cp myclass.cls myclass$n.cls; done
for n in {001..1500}; do cp myclass.cls-meta.xml myclass$n.cls-meta.xml; done
git add .
git commit -m “first commit”
git checkout master
for n in {001..1500}; do cp myclass.cls myclass$n.cls; done
for n in {001..1500}; do cp myclass.cls-meta.xml myclass$n.cls-meta.xml; done
git add .
git commit -m “second commit”
cd .. #back to the sfdx-travisci folder
sudo cp -p $(git diff --name-only branch2 /Users/USERNAME/sfdx-travisci/force-app/main/default/classes) /Users/USERNAME/sfdx-travisci/force-app/main/default/diff
for file in $(sudo cp -p $(git diff --name-only branch2 /Users/USERNAME/sfdx-travisci/force-app/main/default/classes) /Users/USERNAME/sfdx-travisci/force-app/main/default/diff); do if [[ $file == *.cls ]]; then find /Users/USERNAME/sfdx-travisci/force-app/main/default/classes -samefile “$file-meta.xml” -exec sudo cp -p {} /Users/USERNAME/sfdx-travisci/force-app/main/default/diff +; fi; done;

Answer 1

There is a limit to command line length, on most operating systems. The usual solution is not to use a command line, but a pipe, and eventually xargs to get back batches of files as arguments.

For example you can use:

find . | xargs -n 1000 sudo cp ....

in such cases, xargs will split arguments in batches of 1000 files at a time, and it execute sudo cp, at every batch. In this manner you should not have problems (but on very very long directories and filenames).

If you want to execute sudo only once you may want to move find inside sudo (warning: you may have access to more files, as root, and you should care that pipes and redirections are done inside sudo, and not in the calling shell).

Note: you should also change diff to git diff.

Often we use -print0 in find and -0 in xargs, so that you will not have problems with special characters in filenames. (\U+0000 is forbidden in filenames in POSIX compatible systems).

Note: if it works, you may want to send a bug report with the patch to the group who created the original file (if it is not originated by you). We find often such kind of bugs: commands that fails on special cases.