bash script to clone/pull bulk git repo

Question

I've several(22) private repositories on gitlab with https protocol and I want to checkout all repositories providing username and password once from bash script. But can't seem to make it work. Here's how far I got: (any edit/optimization is much appreciated):

#!/bin/bash

read -p "gitlab username: " user
read -sp "gitlab password: " pass
echo
read -p "branch to checkout: " branch
echo

repo[0]='.' #dir
repo[1]='repo.myhost.com/project/app.git' #repo url

repo[2]='plugins'
repo[3]='repo.myhost.com/project/app-core.git'

repo[4]='plugins'
repo[5]='repo.myhost.com/project/plugin1.git'

repo[6]='plugins'
repo[7]='repo.myhost.com/project/plugin2.git'

repo[8]='api-plugins'
repo[9]='repo.myhost.com/project/api-plugin1.git'

repo[10]='api-plugins'
repo[11]='repo.myhost.com/project/api-plugin2.git'

# will add more repo

total=${#repo[*]}
echo "checking out repositories..."
mkdir -p plugins
mkdir -p api-plugins

for (( i=0; i<${total}; i+=2 ));
do
    dir=${repo[$i]}
    trepo="https://$user:$pass@${repo[$i+1]}"
    echo "checking out ${repo[$i+1]} to directory $dir"...
    if cd $dir; then git pull; else git clone -b branch --single-branch $trepo $dir; fi
    echo
done

Edit: My git pull doesn't provide any password, don't know how to do that. Also my password contains @ in it.

The outcome of this script: asking for username/pass again

mamun@linux ~/dev/projects $ ./checkout.sh
git username: myuser
git password: 
project branch: master

checking out repositories...
checking out repo.myhost.com/project/app.git to directory ....
Username for 'https://repo.myhost.com': ^C

What is the error? Also if your `$user` or `$pass` contain special character you must `%`-encode them; `@` is `%40`. — phd, Sep 12 '17 at 04:23
it's asking for username & password again. let me edit the question. — Mamun Sardar, Sep 12 '17 at 04:25
Does cloning `http://$user:$pass@repo.myhost.com/project/app.git` work from command line? — phd, Sep 12 '17 at 05:17
Run your script with "bash -x" and see what's wrong with URL. — phd, Sep 12 '17 at 11:58

score 1 · Answer 1 · answered Sep 12 '17 at 08:35

One easy solution would be to use SSH protocol with SSH key. If the key have a passphrase, you can use ssh-agent.

If you're stuck with HTTP(S), git provides a mechanism to give the username and password without putting them in the URL: GIT_ASKPASS

GIT_ASKPASS is an env-var that points to a script that take one argument: the prompt ("Username for ..." or "Password for ...") and outputs the prompted value.
So a solution is:

GIT_ASKPASS=$(mktemp)
cat <<EOF > ${GIT_ASKPASS}
#!/bin/sh
case "\$1" in
    Username*) echo '${user}' ;;
    Password*) echo '${pass}' ;;
esac
EOF
chmod +x ${GIT_ASKPASS}

# Put here all your git calls that need credentials

rm -f ${GIT_ASKPASS}

VonC · Accepted Answer · 2017-09-12T12:34:08.090

0

As mentioned, a password with @ must be percent encoded (as I mention here).
In pure bash, see this function for instance.

Try in your same bash those commands:

echo git ls-remote http://${user}:${pass}@repo.myhost.com/project/app.git
git ls-remote http://${user}:${pass}@repo.myhost.com/project/app.git

The ls-remote will query the remote repo branches, without pulling or cloning: this is just for testing.

The OP Mamun Sardar confirms in the comments the encoding issue and adds:

Also I made a mistake should use: git clone -b $branch instead of git clone -b branch

I agree and would recommend:

using ${branch} instead of $branch.
checking your script with www.shellcheck.net

edited Sep 12 '17 at 12:34

answered Sep 12 '17 at 05:33

VonC

1,262,500
529
4,410
5,250

ok. I added `pass="${rawpass//@/%40}"` to replace. thanks. Also I made a mistake should use:`$branch` instead of `branch` – Mamun Sardar Sep 12 '17 at 09:49
@MamunSardar Great! I have included your comment in the answer for more visibility, and have added a link to check your script. – VonC Sep 12 '17 at 12:34

bash script to clone/pull bulk git repo

2 Answers2