Managing many git repositories

Question

Setting up a project is easy in git and so I can have separate repository even for small script. Now the problem is how to manage them.

I work in multiple places with these repositories. When I have done changes to some repository, I want to be able to update the repositories in other places.

So I have a directory with many repositories in it.

1. How can I fetch all of them?
2. How can I check whether any of them have uncommitted changes?
3. How can I check whether any of them have changes to merge?

And it would be nice to be able to do these with one command.

The output needs to be silent enough to actually notice the things to do.

Answer 1

I highly recommend the multiple repositories tool mr. I used to use a custom shell script as recommended by others for some time, but using mr has the following benefits for me:

• It's generic: A conjunction of various version control systems can be used, not only git (e.g. Mercurial, SVN, etc.).
• It's fast: mr can execute multiple jobs in parallel. I use a number of git/mercurial repositories and sync them several times a day. Mr tremendously speeds up this process.
• It's easy and quick to manage the list of repository checkouts. Just use 'mr register' rather than modifying the list of projects in your custom script.

Regarding to your question about silent output: The level of verbosity can be modified using the command line switch -q. I prefer the default output which appears to nicely unify the output in a short and clear summary.

I use the following alias for the mr command to ensure that mr always picks up my default project list stored in $HOME, and uses 5 parallel threads:

alias mr='mr -d ~/ -j 5 '

Answer 2

I must say I started with the currently accepted answer (just a bunch of helpers scripts that iterate over the repositories), but all in all, it was a lacking experience for me.

So, after trying mr, repo and git-submodules, I found each lacking in a different way, so, I ended up doing my own variant: http://fabioz.github.io/mu-repo which is a mature tool at this point -- it has workflows which allow you to:

• clone multiple repos
• diff (and edit) current changes in multiple repos
• preview incoming changes
• run commands in multiple repos in parallel
• create groups of repos
• run non git-related commands over multiple repos
• etc (see homepage for more info).

Note that it supports any OS where Python runs ;)

Answer 3

gr (git-run) extends mr's functionality (only for git). I find it easier to organize multiple git repos using its tag system. The code for gr is not well maintained though. If you are using bash, make sure you use it with the -t tag instead of #tag form.

Answer 4

I wrote a commandline tool called gita to manage multiple repos. It shows the status of registered repos side by side, for example

It can also delegate git commands/aliases from any working directory.

Now to answer your questions using this tool:

How can I fetch all of them?

gita fetch

How can I check whether any of them have uncommitted changes?

The gita ll command shows 3 possible symbols next to the branch name, which indicates

• +: staged changes
• *: unstaged changes
• _: untracked files/folders

How can I check whether any of them have changes to merge?

The branch names are colored in 5 ways

• white: local branch has no remote branch
• green: local branch is the same as remote branch
• red: local branch has diverged from remote branch
• purple: local branch is ahead of remote branch (good for push)
• yellow: local branch is behind remote branch (good for merge)

To install it, simply run

pip3 install -U gita

Answer 5

I wrote a tool called "RepoZ" which automatically discovers git repositories as soon as you clone them or change anything in them like switching a branch, for example.

Once found, the repositories are "tracked". That will simply show them in a list of local repositories including a dense status information inspired by posh-git. This contains the current branch and further stuff like file edits and the count of incoming or outgoing commits.

This helps to keep track of your local repositories and unfinished work to commit or push. In addition, you can use the repository list as navigation to switch from one repository to another.

"How can I fetch all of them?"

The version for Windows offers a context menu to fetch or pull a repository. With multi-select you can run actions on multiple repositories at once.

However, you might find another feature very useful:

With Auto fetch, you can tell RepoZ to fetch the remotes of all your git repositories periodically in the background. These fetches won't collide with your local commits, of course. There are no local merge attempts like with git pull.

Answer 6

You could try using repo with a custom manifest.xml file to specify where your repositories are. There is some more documentation on how to do this.

It's the standard for Android AOSP project.

Alternatively you could use git-submodule(1).

Answer 7

I've made an alias and a function to run any git command on all repositories available in a directory (recursively). You can find it here: https://github.com/jaguililla/dotfiles/git

This is the code:

#!/bin/sh
# To use it: source git_aliases

# Example: rgita remote \;
alias rgita='find . -type d -name .git -execdir git'

# Example: rgit remote -vv
rgit() {
  rgita "$@" \;
}

Hope it helps :)

Answer 8

You can use the git-status-all gem for this: https://github.com/reednj/git-status-all

# install the gem    
gem install git-status-all

# use the status-all subcommand to scan the directory
git status-all

There is also a fetch option that will fetch from origin for all the repositories before displaying the status:

git status-all --fetch

Answer 9

I use this script to easily execute git commands in all of my repositories.

#!/bin/sh
if [ ! "$1" = "" ] ; then

   if [ "$GITREPO" = "" -a -d "$HOME/cm/src" ] ; then
      GITREPO="$HOME/cm/src"
   fi

   if [ "$GITREPO" != "" ] ; then

      echo "Git repositories found in $GITREPO"
      echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-"

      DIRS="`/bin/ls -1 $GITREPO`"

      for dir in $DIRS ; do

         if [ -d $GITREPO/$dir/.git ] ; then
            echo "$dir -> git $1"
            cd $GITREPO/$dir ; git $@
            echo
         fi

      done
   else

      echo "Git repositories not found."

   fi
fi

By default the script will look for git repositories in ~/cm/src but you can override this by setting the GITREPO environment variable to your liking.

This script is based on this script.

Answer 10

gitslave is a tool which can run the same command over many repositories by creating a superproject/subproject relationship between the super and the subs. This (by default) provides output summarization so you can concentrate on the repositories which provide unique output (useful for git status, not so useful for git ls-files).

This is typically used for projects where you need to assemble several repositories together and keep them on the same branch or tag at the same time or whatever. For my directory of (bare) repositories I just have a little makefile which lets me run arbitrary git commands, which as you see I primarily use for fsck and gc:

full: fsck-full gc-aggressive
        @:

fsck-full:
        for f in */.; do (cd $$f; echo $$f; git fsck --full || echo $$f FAILED); done

gc-aggressive:
        for f in */.; do (cd $$f; echo $$f; git gc --aggressive || echo $$f FAILED); done

%:
        for f in */.; do (cd $$f; git $@ || echo $$f FAILED); done

Answer 11

For this purpose, you can use every which can run cli commands in multiple directories following this pattern every <command>

Install: npm i -g every-cli

Usage example: every git branch

- repo-1
  master
- repo-2
  develop

Answer 12

If anyone is still looking at this thread please checkout my shell script called gitme

you will need to manually input your local git repos but i use this tool everyday to collaborate multiple git projects on multiple computers.

you can run git clone https://github.com/robbenz/gitme.git

also the script is posted below

#!/bin/bash -e

REPOS=( 
/Users/you/gitrepo1
/Users/you/gitrepo2
/Users/you/gitrepo3
/Users/you/gitrepo4
)

MOVE="Moving to next REPO... \n" 

tput setaf 2;echo "What ya wanna do? You can say push, pull, commit, ftp push, or status"; tput sgr0

read input

if [ $input =  "commit" ]
then
    tput setaf 2;echo "Do you want a unique commit message? [y/n]";tput sgr0
    read ans
    if [ $ans = "y" ]
    then 
        for i in "${REPOS[@]}"
        do
            cd "$i"
            tput setaf 6;pwd;tput sgr0 
            git add . -A
            read -p "Commit description: " desc  
            git commit -m "$desc"
            tput setaf 2;echo  $MOVE;tput sgr0 
            sleep 1
        done 
    else 
        for i in "${REPOS[@]}"
        do
            cd "$i"
            tput setaf 6;pwd;tput sgr0 
            git add . -A
            git commit -m "autocommit backup point"
            tput setaf 2;echo  $MOVE;tput sgr0 
            sleep 1
        done
    fi 
elif [ $input = "push" ] || [ $input = "pull" ] || [ $input = "ftp push" ] || [ $input = "status" ]
    then
        for i in "${REPOS[@]}"
do
    cd "$i"
    tput setaf 6;pwd;tput sgr0 
    git $input 
    tput setaf 2;echo  $MOVE;tput sgr0 
    sleep 1
    done 
else tput setaf 1;echo "You have zero friends";tput sgr0 
fi

I setup an alias in my ~/.bash_profile so alias gitme='sh /path/to/gitme.sh'

Answer 13

I've found a great solution for pulling from the current branch from all sub-directories that have a .git folder, even if each repo has a different branch checked out. The command is a one-liner, flexible enough to modify for different git commands, and can be aliased.

Just copy and paste the following:

find . -type d -maxdepth 2 -name .git -exec sh -c 'cd "$0" && cd .. && git pull origin $(git rev-parse --abbrev-ref HEAD)' {} \;

Breaking it down:

find . -type d -maxdepth 2 -name .git 

Find all directories (-type d) in the current directory (find .) that have the name ".git" (-name .git), looking a maximum of two directories deep (2 rather than 1 because we're looking for the git folder within the git repo folder).

-exec sh -c 

Run the following shell command (exec sh -c)

'cd "$0" && cd .. && git pull origin $(git rev-parse --abbrev-ref HEAD)'

Change directory to the first argument (cd "$0"), then change directory one level up to leave the .git folder (cd ..) then do git pull origin while specifying the branch by running git rev-parse... for the current branch's HEAD commit.

{} \;

The "{}" is the result relative path we get from the initial find command. The ; is used to end the command.

Tested on MacOS 10.14.6 on zsh. As-is, works only when remote and local branches are named the same, AFAIK.

You can modify this to get status. I expect you might be able to add arguments to further make this more flexible but I haven't tried yet.

Answer 14

I wrote a CLI tool for this, see https://manicli.com/. It also has some nice additional features:

• Run commands over many repos
• Declarative configuration
• Flexible filtering, select 1 project, projects having a tag, or a subpath

Answer 15

I have just created a tool that do what you want!

1. How can I fetch all of them? gmaster fetch
2. How can I check whether any of them have uncommitted changes? gmaster status
3. How can I check whether any of them have changes to merge? gmaster status

It is called gitmaster: https://github.com/francoiscabrol/gitmaster

Answer 16

I wrote this simple bash function into my .bash_profile to be able to run git, maven, gradle, or any other bash command in all git repositories only:

# usefull function to work with multiple git repositories
all() {
    failed=0
    printf '>>> START RUNNING: "%s" >>>' "$*"
    for d in */; do
        pushd "$d" > /dev/null
        if [ -d ".git" ]
        then
            printf '\n--------------------------------------------\n\n'
            pwd
            eval $@
            if [ $? -ne 0 ]
            then
                failed=1
                break;
            fi
        fi
        popd > /dev/null
    done
    if [ $failed -eq 0 ]
    then
        printf '\n--------------------------------------------\n'
        printf '<<< RAN "%s" WITH SUCCESS!!! <<<' "$*"
    else
        printf '\n<<< FAILED!!! <<<'
    fi
}

This can be used this way

all git st
all git pull
all ./gradlew clean test
etc.

Answer 17

Disclaimer: I am working on this tool at www.repoflow.com

How can I fetch all of them?
How can I check whether any of them have changes to merge?

• You can fetch all the involved repositories (or push/pull/commit them), after the fetch the UI will be updated with the new repository state, if the repository is diverged you can see the kind of conflicts and begin merge them.

How can I check whether any of them have uncommitted changes?

• On the UI you can see the state of the files for each repository (You can add/remove them individually or in bulk).

I am working on this, but this is also solving the OP questions and help manage multiple repositories in general, you can use the UI or the underlying GraphQL API to create your own scripts, please consider it as another option.

Answer 18

There's a handy tool to fetch all multiple repositories. git-pull-all is a command line tool to execute on multiple git repositories in asynchronously.

Installation

npm install -g git-pull-all

Usage

Assume you have these files and directories:

~/Projects/
  cool-examples/
    .git/
  funny-movies/
  my-todos.txt
  super-express/
    .git/

When you run git-pull-all command on ~/Projects directory, it should find child git repositories (in the above case cool-examples and super-express) then execute git pull on each of them.

$ cd ~/Projects
$ git-pull-all
funny-movies/
Not a git repository
cool-examples/
Already up-to-date.
super-express/
Already up-to-date.
Done!

git-pull-all ~/Projects

---

GitHub link: https://github.com/tatsuyaoiw/git-pull-all

Answer 19

I use Visual Studio code. We have around 20 libraries in our codebase that are all separate Git repos, and the source control tab in Visual Studio Code is pretty good for seeing an "at a glance" view of how far behind or ahead local repos are. There are also settings to periodically fetch to keep that info up to date, as well as a refresh button.

Its not as convenient as a script, but when it comes to source control for me I like to be the one making changes. It takes a very well-written script to be careful to not overwrite changes, etc. With the VS Code approach, you get a good amount of info quickly that you can act on yourself.

Here is a screenshot of what it looks like:

Answer 20

Expanding on https://stackoverflow.com/a/816674/1869000 (Now repo is available on GitHub)

I suggest you check this tool https://github.com/googleapis/github-repo-automation used by Google to manage multiple repos.

About your question about dedicating changes, changes should be PRs so you can react to them.

https://github.com/tj/git-extras could help you write a script for your scenarios.

1. How can I fetch all of them? repo sync on the google solution

2. How can I check whether any of them have uncommitted changes? You can easily write your logic using the lib feature

3. How can I check whether any of them have changes to merge? PR should be the ultimate indication that branches need to merge, but if you have another logic - the above

check git extras commands for ideas see https://github.com/tj/git-extras/blob/master/Commands.md#git-show-unmerged-branches

Answer 21

You should check out rgit on the CPAN which recursively executes git commands on all repositories in a directory tree.

From the docs:

This utility recursively searches in a root directory (which may be the current working directory or - if it has been set - the directory given by the GIT_DIR environment variable) for all git repositories, sort this list by the repository path, chdir into each of them, and executes the specified git command.

Answer 22

I use multi-cursor-select/multi-line editing feature of my code editor (VS code) to create batch/shell script and execute it in one go. With this simple stupid solution, I know what I'm doing and what I can expect from execution of those commands. No guessing, no learning, no customisation needed. Another reason is, every time I want to execute commands on different set of repos which reside in a common parent directory.

Answer 23

Please follow below-mentioned steps to pull code from all the repositories:

1. create a new file using extension as .bat
2. place that file in the same hierarchy where you have all your repository folders
3. edit .bat file and add below-mentioned script

FOR /D %%G in (%cd%\*) Do cd %%G & call git branch & call git pull & cd ..
echo All code pulled successfully!!
pause

4. save the file and double click to start pulling your code

Answer 24

https://github.com/wwjamieson3/envisionTools has a bash script called gitstat that does this and a whole lot more. It's GNU and mac compatible. It can perform fetches from remotes if asked. It shows untracked, modified, added, deleted and staged files. It differences with remotes by showing unmerged commits on remotes and unpushed local commits. It also can display color coded results in summary or detailed mode and even HTML. It uses locate instead of find because it's infinitely faster and will even prompt to update your locate database if it's getting too old.

Answer 25

Looks like writing a script to do it is quite easy. Essentially it needs to iterate over the repositories and then use commands like git ls-files, git diff and git log.