How to split a git repository while preserving subdirectories?

Question

What I want is similar to this question. However, I want the directory that is split into a separate repo to remain a subdirectory in that repo:

I have this:

foo/
  .git/
  bar/
  baz/
  qux/

And I want to split it into two completely independent repositories:

foo/
  .git/
  bar/
  baz/

quux/
  .git/
  qux/  # Note: still a subdirectory

How to do this in git?

I could use the method from this answer if there is some way to move all the new repo's contents into a subdirectory, throughout history.

Answer 1

You could indeed use the subdirectory filter followed by an index filter to put the contents back into a subdirectory, but why bother, when you could just use the index filter by itself?

Here's an example from the man page:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD

This just removes one filename; what you want to do is remove everything but a given subdirectory. If you want to be cautious, you could explicitly list each path to remove, but if you want to just go all-in, you can just do something like this:

git filter-branch --index-filter 'git ls-tree -z --name-only --full-tree $GIT_COMMIT | grep -zv "^directory-to-keep$" | xargs -0 git rm --cached -r' -- --all

I expect there's probably a more elegant way; if anyone has something please suggest it!

A few notes on that command:

• filter-branch internally sets GIT_COMMIT to the current commit SHA1
• I wouldn't have expected --full-tree to be necessary, but apparently filter-branch runs the index-filter from the .git-rewrite/t directory instead of the top level of the repo.
• grep is probably overkill, but I don't think it's a speed issue.
• --all applies this to all refs; I figure you really do want that. (the -- separates it from the filter-branch options)
• -z and -0 tell ls-tree, grep, and xargs to use NUL termination to handle spaces in filenames.

Edit, much later: Thomas helpfully suggested a way to remove the now-empty commits, but it's now out of date. Look at the edit history if you've got an old version of git, but with modern git, all you need to do is tack on this option:

--prune-empty

That'll remove all commits which are empty after the application of the index filter.

Answer 2

I wanted to do a similar thing, but since the list of files that i wanted to keep was pretty long, it didn't make sense to do this using countless greps. I wrote a script that reads the list of files from a file:

#!/bin/bash

# usage:
# git filter-branch --prune-empty --index-filter \
# 'this-script file-with-list-of-files-to-be-kept' -- --all

if [ -z $1 ]; then
    echo "Too few arguments."
    echo "Please specify an absolute path to the file"
    echo "which contains the list of files that should"
    echo "remain in the repository after filtering."
    exit 1
fi

# save a list of files present in the commit
# which is currently being modified.
git ls-tree -r --name-only --full-tree $GIT_COMMIT > files.txt

# delete all files that shouldn't be removed
while read string; do
    grep -v "$string" files.txt > files.txt.temp
    mv -f files.txt.temp files.txt
done < $1

# remove unwanted files (i.e. everything that remained in the list).
# warning: 'git rm' will exit with non-zero status if it gets
# an invalid (non-existent) filename OR if it gets no arguments.
# If something exits with non-zero status, filter-branch will abort.
# That's why we have to check carefully what is passed to git rm.
if [ "$(cat files.txt)" != "" ]; then
    cat files.txt | \
    # enclose filenames in "" in case they contain spaces
    sed -e 's/^/"/g' -e 's/$/"/g' | \
    xargs git rm --cached --quiet
fi

Quite suprisingly, this turned out to be much more work than i initially expected, so i decided to post it here.

Answer 3

Use git-filter-repo This is not part of git as of version 2.25. This requires Python3 (>=3.5) and git 2.22.0

mkdir new_repoA
mkdir new_repoB
git clone originalRepo newRepoA
git clone originalRepo newRepoB

pushd
cd new_repoA
git filter-repo --path foo/bar --path foo/baz

popd
cd new_repoB 
git filter-repo --path foo/qux

For my repo that contained ~12000 commits git-filter-branch took more than 24 hours and git-filter-repo took less than a minute.

Answer 4

This is what I ended up doing to solve this issue when I had it myself:

git filter-branch --index-filter \
'git ls-tree --name-only --full-tree $GIT_COMMIT | \
 grep -v "^directory-to-keep$" | \
 sed -e "s/^/\"/g" -e "s/$/\"/g" | \
 xargs git rm --cached -r -f --ignore-unmatch \
' \
--prune-empty -- --all

The solution is based on Jefromi’s answer and on Detach (move) subdirectory into separate Git repository plus many comments here on SO.

The reason why Jefromi’s solution did not work for me was, that I had files and folders in my repo whose names contained special characters (mostly spaces). Additionally git rm complained about unmatched files (resolved with --ignore-unmatch).

You can keep the filtering agnostic to the directory not being in the repo’s root or being moved around:

grep --invert-match "^.*directory-to-keep$"

And finally, you can use this to filter out a fixed subset of files or directories:

egrep --invert-match "^(.*file-or-directory-to-keep-1$|.*file-or-directory-to-keep-2$|…)"

To clean up afterwards you can use these commands:

$ git reset --hard
$ git show-ref refs/original/* --hash | xargs -n 1 git update-ref -d
$ git reflog expire --expire=now --all
$ git gc --aggressive --prune=now

Answer 5

A cleaner method:

git filter-branch --index-filter '
                git read-tree --empty
                git reset $GIT_COMMIT path/to/dir
        ' \
        -- --all -- path/to/dir

or to stick with just core commands, sub in git read-tree --prefix=path/to/dir/ $GIT_COMMIT:path/to/dir for the reset.

Specifying path/to/dir on the rev-list args does the pruning early, with a filter this cheap it doesn't matter much but it's good to avoid the wasted effort anyway.

Answer 6

If you wish split out just single directory as separate git repository

git-filter-branch has --subdirectory-filter option and it is much simpler then previous mentioned solutions, just:

git filter-branch --subdirectory-filter foodir -- --all

Additionally it change path and place content of directory on top of new repo, not just filter and remove other content.

Answer 7

I used git-filter-repo with filename-callback.

stephen@B450-AORUS-M:~/source/linux$ git filter-repo --force --filename-callback '
  if b"it87.c" in filename:
    return filename
  else:
    # Keep the filename and do not rename it
    return None
  '
warning: Tag points to object of unexpected type tree, skipping.
warning: Tag points to object of unexpected type tree, skipping.
Parsed 935794 commitswarning: Omitting tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c,
since tags of trees (or tags of tags of trees, etc.) are not supported.
warning: Omitting tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c,
since tags of trees (or tags of tags of trees, etc.) are not supported.
Parsed 937142 commits
New history written in 177.03 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at a57e6edb85a3 treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 157
Enumerating objects: 20210, done.
Counting objects: 100% (20210/20210), done.
Delta compression using up to 12 threads
Compressing objects: 100% (17718/17718), done.
Writing objects: 100% (20210/20210), done.
Total 20210 (delta 1841), reused 20038 (delta 1669), pack-reused 0
Completely finished after 179.76 seconds.

It didn't remove empty merge commits, probably due to a bunch of tags that were associated with one side of the tree.

I tried using the top most voted answer, and it didn't seem to remove anything, and took a long time.

Rewrite 3e80e1395bd4f410b79dc0f17113f5b6b409c7d8 (329/937142) (8 seconds passed, remaining 22779 predicted)

22779 seconds = 6.3275 hours