Bash scripting, git and filenames with spaces

Question

I am trying to automate some of my interactions with git using bash scripts. To do so I have a script that will ssh and list all the repositories and then clone them. Part of the script is shown here

DIRS=`ssh $u "ls $p"`;
for DIR in $DIRS; do
    echo "ssh://$u$p$DIR"
    echo "git clone ssh://$u$p$DIR";  
    git clone ssh://$u$p$DIR
done

Example arguments are

-u user@host

and

-p /cygdrive/c/Documents\ and\ Settings/somePath

This work fine for filepaths with spaces in their names up until the git clone part where it throws an error about too many spaces. If I copy and pasted the part echoed by

echo "git clone ssh://$u$p$DIR";

then it clones just fine. I have tried to surround this in quotes but it throws an error

does not appear to be a git repository

What am I doing wrong?

Edit 1

The errors when using git clone "ssh://$u$p$DIR" is

fatal: '/cygdrive/c/Documents\ and\ Settings/somePath/repo1' does not appear to be a git repository
fatal: The remote end hung up unexpectedly

score 3 · Answer 1 · answered Sep 10 '12 at 01:10

If your directory names contain spaces, you most certainly don't want to be doing this:

DIRS=`ssh $u "ls $p"`;
for DIR in $DIRS; do

Since you want to preserve your spaces instead of consuming them as delimiters, the generally accepted way of doing this is telling bash that delimiters are line-feeds:

IFS=$'\n'
for DIR in `ssh $u "ls -1 $p"`; do

If you really must have your variable DIRS, you can use bash's array variables (which will preserve your spaces):

IFS=$'\n'
DIRS=(`ssh $u "ls -1 $p"`)
unset IFS
for DIR in "${DIRS[@]}"; do

Another general form is using read:

ssh $u "ls -1 $p" | while read DIR; do

Or:

while read DIR; do
    ...
done < <(ssh $u "ls -1 $p")

However, these aren't recommended as error handling (especially on ssh becomes a bit more involved when you start using pipes.)

Everything up to this point is just basic bash scripting.

As for this line, you'd want to enclose the second argument with double-quotes because a space in $DIR will make a 3rd argument out of it:

git clone "ssh://$u$p$DIR"

I tried this in git and it seems to work. However as @holygeek said, replacing the spaces with %20 also seems to work, and might beneficial in certain cases. You can do this like so:

git clone "ssh://$u$p${DIR// /%20}"

Alex Brown · Answer 2 · 2012-09-09T20:27:11.050

1

Your problem is almost certainly spaces in filenames.

find and xargs can support a special mode where they separate filenames using \0 (null) characters instead of spaces.

Try this:

ssh $u find $p -depth 0 -print0 | xargs -0 -J % git clone ssh://$u%

You can easily debug a script like the one in your example by running

bash -x myscript.sh

Then it will print out each command line before executing it. That should let you find what it is actually running before the problem.

edited Sep 09 '12 at 20:27

answered Sep 09 '12 at 19:44

Alex Brown

41,819
10
94
108

Using this it seems that three arguemnts are passed '/cygdrive/c/Documents\' 'and\' 'Settings/somePath'. I kind of guessed this is what is happening. I've been trying different combinations of quotes to get the desired effect to no avail. – Codey McCodeface Sep 09 '12 at 20:14

Gilles 'SO- stop being evil' · Answer 3 · 2012-09-10T16:58:51.800

You have two problems. One is that you used ls, which does all kinds of weird things to file names. Don't parse the output of ls. The other is that you didn't put double quotes around $u$p$DIR. Always put double quotes around command substitutions (unless the value of the variable is a whitespace-separated list of glob patterns, which it very rarely is).

Note that in general, $p needs to be quoted as well. Since it's going through ssh, you can't avoid it being interpreted on the other side. If $p does not contain any shell special characters such as spaces, this isn't too difficult. If it does, the easiest way of protecting them is to expand $p on the local side with single quotes around it. The single quotes form a string literal for the remote shell, except where $p contains single quotes. These need to be replaced by the sequence '\''. Bash's parsing rules for the ${p//PATTERN/REPLACEMENT} construct are a bit weird; the two-step process with the intermediate variable $q does the job.

If you can rely on your file names not containing any newline characters, you can print one file name per line on the remote side.

q=\'\\\'\'
ssh "$u" "cd '${p//\'/$q}' && for x in *; do echo \"\$x\"; done" | {
  p=${p%/}
  p=${p//%/%3f}
  case $p in
    /*) :;;
    *) p="/~/$p";;
  esac
  while IFS= read -r DIR; do
    DIR=${DIR//%/%3f}
    git clone "ssh://$u$p/$DIR"
  done
}

Note that some special characters in $p or $DIR will require further quoting in the git ssh: URL. Spaces are ok, but % at least needs to be escaped into %3f. In the code above, I've added support for that, as well as support for the case when $p is a relative path (which will be interpreted relative to the home directory).

I tried to implement this but I found that the `DIR=${DIR//%/%3f}` part returned a single string containing all the filenames. This gave more problems if these names contained spaces. — Codey McCodeface, Sep 10 '12 at 16:45
@medPhys-pl Sorry, there was a mistake in the remote command: I was printing space-separated file names instead of newline-separated. See my fix. — Gilles 'SO- stop being evil', Sep 10 '12 at 16:59

score 0 · Accepted Answer · edited May 23 '17 at 12:30

I've used information from Gilles's and Antak's answer to find a solution as well as some other posts on running ssh on a remote machine and using variables in a bash heredoc.

As suggested by Gilles I avoided using ls and I've used Antak's suggestion of using IFS=$'\n' to avoid multiple arguments from spaces and split arguments on line breaks.

The final code I've settled on is

DIRS=`ssh "$u" << ENDSSH
    cd "$p"
    eval 'for file in *; do echo \\$file; done'
ENDSSH`

IFS=$'\n'
for DIR in $DIRS;
    do
    echo "ssh://$u$p/$DIR"
done
unset IFS

score -2 · Answer 5 · answered Sep 09 '12 at 22:34

-2

See if replacing the spaces with %20 works for you.

answered Sep 09 '12 at 22:34

holygeek

15,653
1
40
50

Bash scripting, git and filenames with spaces

5 Answers5

Linked