Git alternative to sending out tarballs and asking patches be sent back for a sub-project?

Question

I have a largeish project which has within it a sub-directory which itself is a project.

I want to manage this sub-directory within the larger project sharing the same branches, tags, etc, for simplicity sake. The integrity of the larger project and ability to track it as a monolithic git repo is very important to me.

But I also want to publish and accept pull requests from contributors in the smaller project (sub-directory), without exposing them to my larger project. The sub-project requires some ancillary files (like an Eclipse .project) in its parent in order to be fully self-contained.

I've looked into submodule, subtree merge, and sparse-checkout, but couldn't figure out how to do this. As a stop-gap measure I just ship out a tarball of the sub-directory and ask for patches in return.

Any ideas how to improve this with a git-centric workflow?

Answer 1

Ok, that is not a perfect setup, but should be good enough.

The idea is to have one branch in which everything apart from that subdirectory and common files is removed (I mean: don't exist at all), and to accept pull requests for that branch (you may create a new repository containign only that branch).

It is a bit twisted, so I'm going to show it through an example. Let's say, your project was created with commads like this:

mkdir bigproj
cd bigproj
git init
echo "common file" >common
mkdir subproj
echo "subproj content" >subproj/content
mkdir other
echo "other content" >other/content
git add common subproj other
git commit -m 'Initial commit'
git commit --allow-empty -m 'Some history'

It has a common file inside, a subproj subdirectory with some content, and the other subdirectory with another content. Tree:

.
├── common
├── other
│   └── content
└── subproj
    └── content

Now let's create a branch containing only common and subproj without history:

git checkout --orphan subproj-branch
git rm -rf .  # clear the index
git checkout master -- common subproj  # put `common` and `subproj` back to index
git commit -m 'Initial commit for subproj-branch'

Resulting tree:

.
├── common
└── subproj
    └── content

Merge this branch back into master to avoid possible false conflicts:

git checkout master
git merge subproj-branch  # obviously no conflicts

Now we can publish subproj-branch in some dedicated repository:

git remote add subproj-repo <some url>
git push subproj-repo subproj-branch:master --set-upstream  # -f may be needed.
# And remote branch doesn't have to be named master, of course.

Repository is published, we got some patches. Now we can merge them:

git checkout subproj-branch
git pull
git checkout master
git merge subproj-branch

That is the base flow, which allows making changes in subproj-repo and incorporating them into main repository. Now, making it the other way around is a bit more problematic, but possible. There are to possibilities:

1. Changes touch only subproj/common. We may take them "as they are":

   git checkout subproj-branch
   git cherry-pick master  # replace master with anything you actually need
   git checkout master
   git merge subproj-branch
   git push subproj-repo subproj-branch:master
   

2. Changes touch both subproj/common and other files. You can manually checkout each changed file into subproj-branch then commit and merge back to master (to avoid false conflicts in future). That is not perfect, and you may want to alter that step somehow.

   git checkout subproj-branch
   git checkout master -- common subproj
   git commit -m 'Some changes'
   git checkout master
   git merge subproj-branch
   git push subproj-repo subproj-branch:master
   

Important part here is merging changes back to master. That may seem nonsensical, but may prevent some conflicts from happening.

Whoa, that is a long answer. I hope it will be helpful :P

Answer 2

This is not exactly a full answer, but working from @Frax's answer I ran into a few things I didn't understand, or didn't work as I expected, and I needed to modify it as follows:

• First I used filter-branch to create a new branch that has only the contents of the common subdirectory with history. (* I avoid making this a separate repo because I have a limit on number of private repos at GitHub *). This will simplify all remaining operations as most git commands work naturally at a branch scope, but I keep getting confused when trying to do things at a subtree scope.
• Even this filtered history is extremely bloated so I proceeded to create a branch with no history using Frax's suggested --orphan option. Excellent suggestion.
• It's important to do a git merge back onto the full branch (not the orphaned branch), this merge direction seems to be backwards in Frax's answer, or I just don't understand what I'm doing. The result is that the commit hash from the new branch can be seen in the history branch.
• Finally, use submodule add –b branch to incorporate this new branch into both the original project and subproject.
• After just a few minutes of trying to use this, I already ran into limitations with Eclipse's handling of submodules. Looks like next alternative is to just split up the repos.

A few more notes

• This question appears to be very similar: How can I keep a subfolder of a git repo in sync with a subfolder of another git repo - I just offered 100 point bounty on that question
• Using sparse checkout to grab a subdirectory from one repo and commit to the other "almost works", except, my entire 500MB repository gets replicated behind the scenes even though my sparse checkout covers only a handful of text files.
• Using --depth=1 was hopeful, except trying to commit to the 2nd repo I got ! [remote rejected] master -> master (shallow update not allowed)
• Another similar question (unanswered): Child git repository as subset of a main repository
• The answer from @Frax below works fine if I push to a fresh new branch on the 2nd remote. But I wanted it to be integrated with the remote master so that additional supporting files that are necessary to run the sub-project in stand-alone mode could be in the same branch.

Answer 3

I found the answer hidden at How do I merge a sub directory in git?

The key bit of git magic is to use the following to sync up the two common subdirectories:

git read-tree --prefix=MyHugeProprietaryWebApp/public_html/ -u contrib/master:MyOpenSubproject/public_html/

Where

• MyHugeProprietaryWebApp is the project's top-level directory below the repository root (i.e. this is the project root folder in Eclipse)
• public_html is the subdirectory containing the code I want to work with contributors on (in my specific case this is some PHP layout code). It apperas twice in the command because the subdirectory is named the same in both repositories.
• contrib is the repository on github (shared with contributors) which I created earlier using git checkout --orphan and git push as suggested by Frax; I have not tested possible alternatives such as --depth=1.
• MyOpenSubproject is the root folder of the smaller project and is what contains the Eclipse .project and other ancillary files that make the sub-project self-contained. These ancillary files are NOT shared with the larger project and includes extra documentation, tests, etc, that are only relevant to the outside contributors and not the insiders working on the larger project.

My experience with this approach has been limited to a few dry-run tests, but I'm happy with what I see so far. I have not yet experimented with pull -s subtree -X path which I may need at some point.