Create and push EXACT git repository containing submodules to github with EXACT folder structure

Question

I think mine is the simplest use case of git submodules possible.

I have a directory structure

<project-name>
 ---<directory1>
 ---<directory2>
 ---<directory3>
        ---<directory3.1>

Each sub directory is a git repository. I just want to track different git repositories added in my <project-name> directory, which is my main (or super) repository.

I have figured out how to add each of these git repositories as submodules in my super repository, which is also a directory which contains them. In local I have exactly what I want.

So some of the git submodules are two depths inside my super repository, as long as I use

git submodule add <relative-path-to-submodule> <relative-path-to-submodule>

It takes care of it just fine.

Why in the world I would like to do this? Because this represents the conventional local directory structure in the CFD framework (OpenFOAM), I am developing my code for.

I and others develop different submodules seperately and it is absolutely valuable to track each submodule/subdirectory separately.

The goal is to let my end users to clone my code repository, change the name and start compiling and using the code. (Building is super easy and by convention)

I just can't figure out how to push all the code to github, including the actual files from the
submodule as files in that particular subdirectory.

Obviously when I push to github, the submodule just appear as commits, no actual code from that subdirectory is pushed.

I don't (can't afford to) want to add subdirectories in the super repository. All answers tell me to do that, which is pointless.

I have found the exact question, which is unanswered (Git: Push a local git submodule, including submodule files, to a remote bare repository)

Answer 1

I just can't figure out how to push all the code to github, including the actual files from the submodule as files in that particular subdirectory.

First, don't use submodules, they are not a good fit for what you want to do.

Second, try the subtree approach: I explained the difference between submodule and subtree, and illustrated subtree here.

See:

• "The power of Git subtree" by Nicola Paolucci.
• "Git subtree: the alternative to Git submodule"

git subtree lets you nest one repository inside another as a sub-directory. It is one of several ways Git projects can manage project dependencies.

Answer 2

Here is a step by step solution for reproduction. This is how I make all my submodule workflows work.

The involved mininmal example repositories are linked here for those who are interested:

https://github.com/jankapunkt/submodule-main

https://github.com/jankapunkt/submodules-sub1

Part I - adding the submodule

1. Create a new bare repository, which will be our main dev repo. Let's call ist yourName/submodule-main

2. Create a second new repository with some folders and content, which will be our submodule repo. Let's call it yourName/submodules-sub1

3. Clone the sub1 repo, add some content and push it back to origin

git clone git@github:yourName/submodules-sub1.git
cd submodules-sub1/

# create some content
mkdir somefolder
touch HELLO.md
echo "yeehaw" > ./HELLO.md

# add, commit and push
git add somefolder/
git commit somefolder/HELLO.md -m "added some content"
git push origin master
# note the commit hash here: 
# dbcc1ef..8293526  master -> master

4. Go back and delete the repo (or not, doesn't matter)

5. Clone the submodule-main repo

cd ..
git clone git@github.com:yourName/submodule-main.git

6. Add the sub1 as submodule with tracking a specific branch (in our case master branch):

cd ./submodule-main/
git submodule add -b master git@github.com:jankapunkt/submodules-sub1.git ./submodules-sub1

Note, that the pattern is submodule add -b <branch> <repository> <path>. The now created structure looks like this:

.
..
.git
.gitmodules
LICENSE
README.md
submodules-sub1

And .gitmodules should contain

[submodule "submodules-sub1"]
    path = submodules-sub1
    url = git@github.yourName/submodules-sub1.git
    branch = master

7. Add / commit these changes

git add -A
git commit -a -m "sub1 submodule added"
git push origin master

Go check your repo on Github, it will contain an exact copy of the current development state of sub1 including the reference to the latest commit, for example:

submodules-sub1 @ 8293526

It should contain a complete copy / snapshot of the referenced commit of this submodule including all files and folders.

Part II - updating the submodule

Now you want to always update your main repo with the tracking state of the sub repo.

1. go into the submodule directory and make some changes

cd submodules-sub1/somefolder/
touch FOO.md
echo "bar baz" > FOO.md
git add FOO.md
git commit FOO.md -m "added FOO.md"

2. Push your state to the branch, which is tracked by the submodule-main repo

Now here are two cases that can happen. Enter git status to see your current HEAD position:

a - you are already tracking master

On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

Great, you are already tracking the right branch, just git push origin master

b - you are in detached HEAD state (for example after submodule-main did a submodules update)

Now you need to make HEAD point to the right commit (I use some example hash here):

git log -1 # copy the commit hash, for example e6227de6b7042c06afa0062ec900386a817a058e
git checkout master
git reset --hard e6227de6b7042c06afa0062ec900386a817a058e
git push origin master

3. Update main repo

Now you have updated the code in your sub1 directory but nothing has been updated in the main yet. Therefore we also need to update the main repo and point to the latest submodule commits:

cd ..
git status

you should see a notice about new commits:

Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   submodules-sub1 (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

Now you commit these changes, too and push all to your origin:

git commit submodules-sub1/ -m "sub1 updated some folder content"
git push origin master

Note, that on your origin there is not another commit pointed to in your main repo:

submodules-sub1 @ e6227de

Part III - Cloning for your endusers

Assume your endusers want now clone a kind of "snapshot" where all submodules point to a certain commit on a certain branch (I really simplified this here using master with the latest commits) they should just clone recursive and should be able to build immediately:

git clone git@github.com:jankapunkt/submodule-main.git --recurse-submodules

You can also verify this by entering the submodules-sub1 folder and enter git status, which should now point to e6227de:

HEAD detached at e6227de
nothing to commit, working tree clean

Used documentation is

https://git-scm.com/docs/git-submodule

https://git-scm.com/docs/git-clone

Answer 3

Try using git subtree instead of git submodule

It sounds like submodules are not the best way to do what you are trying to do. I would reccommend using subtrees instead. Subtrees are similar to submodules, but have some differences that would work better for your use case.

Here is a decent explanation of how git subtree works:

https://www.atlassian.com/git/tutorials/git-subtree

In short, subtrees are similar to submodules in that they allow you to add one repo as a subdirectory of another, optionally keeping track of remote references and commit history.

When you use subtree, all of the source code from the included repo will be included in your own repo. All someone will have to do to clone all of the code in your repo and the included repos is to clone your repo. No additional setup steps are required.

Subtrees also make it easy to modify the code of the included repos and push the changes back to the original repository.

Here is an example command for adding a submodule:

git subtree add --prefix <subdir> <other repo> <branch of other repo>

You can't do what you want to do with submodules

Git submodules do not store the code of the other repositories in your git repository. Instead, they store a reference to the other repository in the .gitmodules file. You will be able to see the .gitmodules file in your repository. That is the only capacity in which the sub repositories exist in your repository when you use submodules. The code is not in your repository.

https://git-scm.com/book/en/v2/Git-Tools-Submodules

The DbConnector directory is there, but empty. You must run two commands: git submodule init to initialize your local configuration file, and git submodule update to fetch all the data from that project and check out the appropriate commit listed in your superproject

Answer 4

Submodules allow you to keep a Git repository as a subdirectory of another Git repository. This lets you clone another repository into your project and keep your commits separate. - git-scm.com

Because of this, what you are trying to do is against submodules design in git. At the time you want to track the files into your repo, it means you want to have the commits together. However, according to your goal. You don't need to push the exact folders to your "main" repository.

When you add submodules a file .gitmodules is created and track teh different sources you need in this current repository. This file will get pushed to you repository (same as .gitignore for example) and at the time someone clone it just needs to use the command: git clone --recurse-submodules https://github.com/chaconinc/MainProject and it will get not only your code but the ones from each of the submodules tracked on .gitsubmodules file.

Reference: https://git-scm.com/book/en/v2/Git-Tools-Submodules

Answer 5

I just can't figure out how to push all the code to github, including the actual files from the submodule as files in that particular subdirectory.

As you surely know : the super repository can "see" the content of a submodule only through its commits.
This means you have to commit all submodules individually before you can commit a consolidated version of your super repository :

cd directory1
git add <files...>
git commit
git push    # if you want to *share* the content with others, you additionally need to 
            # publish the content of this subrepo (aka : git push)
cd ..

cd directory2
git add <files...>
git commit
git push
cd ..

# etc ...
# and finally, from root folder :
git add *
git commit
git push

Note that the same rules apply to directory3 :
if directory3.1 is a submodule of directory3, you need to commit the content of directory3.1 before committing directory3.

You may write a script which will scan through each submodule and run said actions, which can reduce the amount of clerical work to do, but these actions need to be applied.

---

If you want to have the ability to treat your root repository more like a regular repo with "files you see upfront" instead of submodules, I would second VonC's advice : look at how you could use git subtree instead.

It does come with its own list of extra tasks (for updates, pushing to sub repos, merging ...),
but it also enables you to view and commit the files of the main repo without being forced to follow the modules hierarchy.