Break large repo into multiple smaller repos on push

Question

I have a project hosted on github with the following structure

github.com/example/allpackages

.
├── .git
└── packages
    ├── example-1
    ├── example-2
    └── example-3

On each push to github I would like for the contents of each package to pushed to a repo the corresponds to the repo name e.g.

github.com/example/example1

.
├── .git
└── example1

github.com/example/example2

.
├── .git
└── example2

etc.

Any one have any ideas on how to automate this? Someone mentioned using Travis-CI for this task, but I couldn't find any specifics on how that might work.

My ideal solution:

Would be complete automated -- meaning the update would not be triggered by the pushing computer, but rather github or an external service.
Migrate the history from "allpackages" to the corresponding subrepos on update

Any guidance on where to start researching would be greatly appreciated. Thanks in advance!

Edit:

@VonC suggested using submodules, and making commits using git submodule foreach --recursive

Pros:

Dry - the code for each submodule lives in one place.
Able to track changes to submodules in parent.
Using git submodule foreach --recursive using an alias allows for a single commit to multiple submodules

Cons:

Viewing the changes made to the submodules is not immediately clear. One must dig into the submodule to see what changes had been made.
Using git submodule foreach --recursive is cumbersome, and not as elegant to work with as a regular commit.

For this particular use case. The "package" repos e.g. github.com/example/example1 will be read only in a sense. I will not be pushing to them directly. They will only receive updates when allpackages is updated. The only reason that they need to be created, is because the package manager that utilizes them requires separated repos for each package.

score 3 · Accepted Answer · edited May 23 '17 at 12:29

3

If you declare all your exampleX folders as git repos, and make them submodules of the parent repos allpackages, then:

each exampleX repo can have its own upstream repo on GitHub
(since git 1.7.11), you can push everyhting to their respective repos
```
cd allpackages
git push --recurse-submodules=on-demand
```

One command, and everything is pushed.

edited May 23 '17 at 12:29

Community

1
1

answered Oct 06 '13 at 08:16

VonC

1,262,500
529
4,410
5,250

Thanks @VonC. Is there a way to commit to multiple submodules at once? The main reason I want to have the packages under *allpackages* is because I make changes that affect multiple packages in a single commit. – Kyle Finley Oct 06 '13 at 13:41
@KyleFinley you mean, add and commit *within* multiple submodules? Because after modifying the history of a submodule, you still need to go back to the parent repo, add and commit there (and that would register the new SHA1 of *all* modified submodule in one step) – VonC Oct 06 '13 at 13:53
@KyleFinley But if you do meant add and commit *within* multiple submodules, then you can try http://stackoverflow.com/a/14856390/6309. – VonC Oct 06 '13 at 13:55
Correct. I would prefer to do a single add and commit, even if the files span multiple submodules. I've considered using the foreach command, but I found it cumbersome to work with. I think my ideal solution would allow me to treat *allpackages* as a single repo, and not worry about how the packages get redistributed. I'll update my question to explain this better. – Kyle Finley Oct 06 '13 at 14:02
@KyleFinley a simple git alias would take care of the cumbersome command for you. If you can make a series of shell command work, you can call them from a git alias. http://stackoverflow.com/a/1309674/6309. So I would maintain my foreach solution here. It is the "submodule-way" to go. – VonC Oct 06 '13 at 14:19
absolute, git alias would simplify the commits. I've updated my question with information on my motivation. It wasn't clear initially, but the submodules are secondary to me. I want things to be as simple as possible when dealing with "allpackages" and then create the submodules as "read only" reps. Your answer and comments have helped me understand the "submodule-way" better. If I can't find a solution that hides more of the complexity I'll mark this as accepted. Thanks again! – Kyle Finley Oct 06 '13 at 14:44

score 2 · Answer 2 · edited May 23 '17 at 12:21

Github won't run repo hooks. Setting up a github proxy repo with a post-update hook that adapts the auto-tracking in this answer and then forwards the pushes will, as I read it, satisfy the stated requirements.

That said, though, I'm with VonC. If you don't go with submodules you're going to be stuck with something like that -- and really, it's nothing more than fragile, crippled, stealth submodules that only work for that one repo, guaranteed to break as soon as someone pushes anywhere but to that proxy. It can be very convenient for your own work, in a private repo, and there are ways to kick the fragility one step farther down the road, but for a shared discipline it's not nearly robust enough.

For shared work I'll bet you wind up sooner or later switching to submodules anyway, and discovering they're not very mysterious. So ignore this and accept VonC's answer :-)

score 1 · Answer 3 · edited May 23 '17 at 10:25

1

I still prefer the clean "submodule" approach I recommended earlier.

But one other more hackish way to push one folder of a repo into an independent upstream repo would be to use the nested repo technique:

cd myrepos
git clone https://github.com/example/example1
git clone https://github.com/example/allpackages
cd allpackages/packages/example-1
git --git-dir=../../../example1/.git add .
git --git-dir=../../../example1/.git commit -m "commit for example1"
git --git-dir=../../../example1/.git push

That would consider, for the example1 GitHub repo, the working tree of allpackages/packages/example-1: --git-dir makes Git believes that allpackages/packages/example-1 is a (nested) git repo of its own, with a remote 'origin' referring GitHub.

But: that would mean 3 different push commands, unless you wrap those commands in a script and call that scripts through an git alias:

git config alias.pushall '!sh pushall.sh'

edited May 23 '17 at 10:25

Community

1
1

answered Oct 07 '13 at 20:53

VonC

1,262,500
529
4,410
5,250

Thanks again @VonC I wasn't aware of `--git-dir=`, I'll experiment with it. I don't mind doing multiple pushes (as you said this can be automate this). The more I think about it, the feature that I really want is one repo with the complete commit history from all packages -- will technique provide that? – Kyle Finley Oct 07 '13 at 21:16
@KyleFinley not quite, because it would force you to add and commit first, like those were independent repo, even though you already add and commit that same content (but as part of the allpackages repo). I have amended the answer to make that step more visible. So you end up with two different histories: one for allpackages, one for the examples repos. – VonC Oct 07 '13 at 21:19
@KyleFinley This is different for the submodules, where the commits you do in a parent repo only register the latest SHA1 of the submodule in which you have made new commits. There is no history duplication there. – VonC Oct 07 '13 at 21:20
interesting, I'll experiment with is; it sounds like it might be closer to what I'm looking for. Thanks! – Kyle Finley Oct 07 '13 at 21:23

Break large repo into multiple smaller repos on push

3 Answers3