Trying to correct my GitHub submodule to remove it and add to the root folder for commits

Question

I'm trying to setup my git root folder to simply commit all changes to github when I git add . then git commit -m "a simple message" and git push. The problem is that I've somehow made my gatsby folder into a submodule and I can't seem to remove that and revert back to the gatsby folder just being a part of the root folder that needs to be committed. Here's my most recent Terminal activity:

ericphifer@Erics-MacBook-Pro WestWater % git add .
ericphifer@Erics-MacBook-Pro WestWater % git commit -m "update files and folders to align between github and vscode"
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
        modified:   gatsby-ww (modified content, untracked content)

no changes added to commit (use "git add" and/or "git commit -a")
ericphifer@Erics-MacBook-Pro WestWater % mv WestWater westwaterco
mv: rename WestWater to westwaterco: No such file or directory
ericphifer@Erics-MacBook-Pro WestWater % git rev-parse --show-toplevel #print repo top-level directory
/Users/ericphifer/Desktop/WestWater
#print
fatal: ambiguous argument '#print': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
ericphifer@Erics-MacBook-Pro WestWater % git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
        modified:   gatsby-ww (modified content, untracked content)

no changes added to commit (use "git add" and/or "git commit -a")
ericphifer@Erics-MacBook-Pro WestWater % git commit -a
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
        modified:   gatsby-ww (modified content, untracked content)

no changes added to commit (use "git add" and/or "git commit -a")
ericphifer@Erics-MacBook-Pro WestWater % git conig --list
git: 'conig' is not a git command. See 'git --help'.

The most similar command is
        config
ericphifer@Erics-MacBook-Pro WestWater % git config --list
credential.helper=osxkeychain
push.default=current
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=https://github.com/EricPhifer/westwaterco
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
submodule.gatsby-ww.url=https://github.com/EricPhifer/westwaterco.git
submodule.gatsby-ww.active=true
ericphifer@Erics-MacBook-Pro WestWater % git submodule deinit
fatal: Use '--all' if you really want to deinitialize all submodules
ericphifer@Erics-MacBook-Pro WestWater % git submodule deinit --all
ericphifer@Erics-MacBook-Pro WestWater % git config --list
credential.helper=osxkeychain
push.default=current
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=https://github.com/EricPhifer/westwaterco
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
submodule.gatsby-ww.url=https://github.com/EricPhifer/westwaterco.git
submodule.gatsby-ww.active=true
ericphifer@Erics-MacBook-Pro WestWater % rm -rf submodule.gatsby-ww.url=https://github.com/EricPhifer/westwaterco.git
ericphifer@Erics-MacBook-Pro WestWater % git config --list
credential.helper=osxkeychain
push.default=current
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=https://github.com/EricPhifer/westwaterco
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
submodule.gatsby-ww.url=https://github.com/EricPhifer/westwaterco.git
submodule.gatsby-ww.active=true
ericphifer@Erics-MacBook-Pro WestWater % rm -rf .git/modules/a/submodule
ericphifer@Erics-MacBook-Pro WestWater % git rm -f a/submodule
fatal: pathspec 'a/submodule' did not match any files
ericphifer@Erics-MacBook-Pro WestWater % git config --list
credential.helper=osxkeychain
push.default=current
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=https://github.com/EricPhifer/westwaterco
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
submodule.gatsby-ww.url=https://github.com/EricPhifer/westwaterco.git
submodule.gatsby-ww.active=true
ericphifer@Erics-MacBook-Pro WestWater % mv submodule.gatsby-ww submodule.gatsby-ww-temp
mv: rename submodule.gatsby-ww to submodule.gatsby-ww-temp: No such file or directory
ericphifer@Erics-MacBook-Pro WestWater % mv gatsby-ww gatsby-ww-temp
ericphifer@Erics-MacBook-Pro WestWater % git submodule deinit -f -- gatsby-ww
ericphifer@Erics-MacBook-Pro WestWater % rm -rf .git/modules/gatsby-ww
ericphifer@Erics-MacBook-Pro WestWater % git rm -f gatsby-ww
rm 'gatsby-ww'
ericphifer@Erics-MacBook-Pro WestWater % git config --list
credential.helper=osxkeychain
push.default=current
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=https://github.com/EricPhifer/westwaterco
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
submodule.gatsby-ww.url=https://github.com/EricPhifer/westwaterco.git
submodule.gatsby-ww.active=true
ericphifer@Erics-MacBook-Pro WestWater % mv gatsby-ww-temp gatsby-ww
ericphifer@Erics-MacBook-Pro WestWater % git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        deleted:    gatsby-ww

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        gatsby-ww/

ericphifer@Erics-MacBook-Pro WestWater % git add .
warning: adding embedded git repository: gatsby-ww
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint: 
hint:   git submodule add <url> gatsby-ww
hint: 
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint: 
hint:   git rm --cached gatsby-ww
hint: 
hint: See "git help submodule" for more information.
ericphifer@Erics-MacBook-Pro WestWater % git commit -m "add gatsby-ww for commits"
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
        modified:   gatsby-ww (modified content, untracked content)

no changes added to commit (use "git add" and/or "git commit -a")
ericphifer@Erics-MacBook-Pro WestWater % 

Much of this one is attempting to follow the directions of this thread How do I remove a submodule?, but you can see that every time I check the git config --list the submodule is always still there. This is one of many stackoverflow threads I've tried to follow and nothing is working, so this is my last ditch effort to understand what's wrong and how to fix it. Git is so frustrating for me, please help and thank you.

Just checked another project and it's having the same issue except that it doesn't have a submodule for the gatsby file:

ericphifer@Erics-MacBook-Pro Joseph Center Code % git config --list
credential.helper=osxkeychain
push.default=current
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=https://github.com/EricPhifer/joseph-center.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
ericphifer@Erics-MacBook-Pro Joseph Center Code % git status 
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
        modified:   gatsby-jc (modified content)

no changes added to commit (use "git add" and/or "git commit -a")

Answer 1

First, let's get this out of the way, as it's relatively minor:

submodule.gatsby-ww.url=https://github.com/EricPhifer/westwaterco.git
submodule.gatsby-ww.active=true

These are settings that Git copied from the .gitmodules file, describing the submodule gatsby-ww, to your .git/config. Git does this at various points when cloning superprojects and submodules because the .gitmodules file contains the original URL but you might wish to alter this URL without affecting the .gitmodules file. (That's the .url specifically; the .active setting just tells other parts of Git that the submodule is actively in use.)

We can therefore ignore these: they're only used if/when the submodule is in active use in the first place, and it's the fact that the submodule is in active use that is your stumbling block.

Once you've fixed that—assuming that fixed is the right word—you can use git config --unset submodule.gatsby-ww.active and git config --unset submodule.gatsby-ww.url to remove these two settings, or git config --edit to run your configured editor on the .git/config file (which is a Git-ized flavor of an INI file; it's usually pretty clear how to tweak these in any text editor).

Just checked another project and it's having the same issue except that it doesn't have a submodule for the gatsby file ...

This other project does have a submodule, though it's named gatsby-jc, not gatsby-ww.

Anyway, let's go back to your attempt to de-submodule gatsby-ww in the first project:

ericphifer@Erics-MacBook-Pro WestWater % mv gatsby-ww gatsby-ww-temp
ericphifer@Erics-MacBook-Pro WestWater % git submodule deinit -f -- gatsby-ww
ericphifer@Erics-MacBook-Pro WestWater % rm -rf .git/modules/gatsby-ww
ericphifer@Erics-MacBook-Pro WestWater % git rm -f gatsby-ww
rm 'gatsby-ww'

These commands were fine, on their own. You:

• moved the existing Git repository in gatsby-ww out of the way: this Git repository will continue to exist;
• used git submodule deinit to clear out the existence of gatsby-ww as a submodule;
• removed any clone that existed in .git/modules/gatsby-ww; and
• removed the index entry giving gatsby-ww its existence as a gitlink.

We'll get back to this last detail in a bit, as this has a lot to do with how submodules work.

Things went very wrong here though:

ericphifer@Erics-MacBook-Pro WestWater % mv gatsby-ww-temp gatsby-ww
ericphifer@Erics-MacBook-Pro WestWater % git add .
warning: adding embedded git repository: gatsby-ww

Note the warning: adding embedded git repository ... message here. I've trimmed away the verbose hint below it, but this step put gatsby-ww back as a submodule. You must not git add the gatsby-ww directory as it stands at this point, because it contains a .git.

What you need to know about submodules

Submodules are Git's way of letting one Git repository refer to another Git repository. By design, no Git repository can ever contain another Git repository. This is simply forbidden—for security purposes, mainly, although the binary files that Git stores inside its databases are automatically ill-suited to be stored in another Git repository, so even if it weren't already forbidden, it would be a bad idea.

Since a repository literally can't contain another repository, but people would like to compose or chain repositories, Git has a "solution" (sort of) here: the submodule. A submodule is, in part, a declaration of the form: After you've cloned this repository, and checked out some particular commit, you—Git—should clone another repository. To clone some other Git repository, Git needs to know the URL of that repository. Having made that clone, Git then needs one more pair of items: the raw hash ID of some commit within that repository that Git should git checkout in the sub-repository, and the path at which that checkout should live as a submodule.

Git calls the "outer" repository the superproject and the "inner" (sub) repository the submodule. These unfortunately both start with the letter s, so I use R and S to denote superproject Repository and Submodule respectively.

The first part, and half of the second, go in the .gitmodules file. This lists the URL and the path. Half of the second part, and the rest of the second part, go in each commit in R. Each commit in R contains a <pathname, hash-ID> pair. This pair takes the form of what Git calls a gitlink.

A normal file, in a Git repository, is an entity with mode 100644 (regular, non-executable file) or 100755 (executable file). A symbolic link, in a Git repository, is an entity with mode 120000. You can see these things in git ls-tree -r output:

...
100755 blob 9c125f298aea6863a9370c16a0f70b7cccc73b2d    GIT-VERSION-GEN
100644 blob 66389ce05915d776e5c8ff49aea40e20bc40eb62    INSTALL
100644 blob d38b1b92bdb2893eb4505667375563f2d6d4086b    LGPL-2.1
...
100644 blob c3565fc0f8f7df81232deca663bf55e14737ae97    Makefile
100644 blob eb8115e6b04814f0c37146bbe3dbc35f3e8992e0    README.md
120000 blob f071367dc30cd9c12cc010539f4ca151d1282757    RelNotes
...
100644 blob 6102893fccb96cf2203b524664c3b2c88c84d9b1    builtin/log.c
100644 blob 45cc3b23dd6845d902e64997a80c7c7347110c6d    builtin/ls-files.c
100644 blob 1794548c71179afb68c74a231749be5f5fdae3ee    builtin/ls-remote.c
100644 blob 3a442631c71a07f0d0dda791ed72af83f2b92bca    builtin/ls-tree.c
...

The above is a snippet of output from git ls-tree -r on a commit in the Git repository for Git. Note how the file GIT-VERSION-GEN is executable and RelNotes is a symbolic link.

A submodule in a commit is much like any other file, except that it has mode 160000. The big ugly hash ID that comes after the type (commit) is that of some commit in the submodule:

160000 commit 855827c583bc30645ba427885caa40c5b81764d2  sha1collisiondetection

The commit itself, and hence all of the files, are not stored in the superproject: repository R contains only the .gitmodules file—as many copies of it as needed, one per commit minus all the usual de-duplication of identical files—plus the gitlinks, one per commit. Here is the contents of the .gitmodules file:

$ cat .gitmodules
[submodule "sha1collisiondetection"]
        path = sha1collisiondetection
        url = https://github.com/cr-marcstevens/sha1collisiondetection.git
        branch = master

(Since this file is the same in most of the commits, there's really just the one shared copy of it. Look at the output from git ls-tree -r HEAD, or any other commit, to see the hash ID of the blob holding the above text.)

What this means for you

If you'd like the files in your gatsby-ww directory to be placed into the next commit you make in R, you must remove the .git that is in gatsby-ww/.git. This could be a .git directory, holding the entire repository S, or it could be a file named .git, holding the path where Git expects to find S. Either way, the fact that it exists means that git add will refuse to add the files. Instead, git add will detect that gatsby-ww is a repository, and will add the appropriate gitlink for S instead of adding the files. Since this doesn't create-or-update .gitmodules correctly, git add will also print out a great big warning message about how the submodule it just added is kind of half-assed and probably won't work right for anyone else (including yourself-in-the-future).

Remember that a repository is a collection of commits. It is not a collection of files! The files in some repository are in the commits, but each commit has every file (albeit compressed, Git-ified, and de-duplicated into a binary format that should not be stored in another Git repository). You cannot put a Git repository into a another Git repository, except by trickery (temporarily renaming .git for instance) and in general you should not do that (because it works poorly). If you are using submodules, think about whether you want to use submodules; if you do, keep using them; if not, think about what you do want to keep.

If you do want to use a submodule, and you have made changes in the working tree in the submodule, you must:

• enter the submodule—it's a Git repository of its own—and add and commit the changes;
• git push those changes to the place that others will git clone from; and then
• exit the submodule and git add the path to S so that Git can put a new, revised gitlink into the next commit, using the hash ID of the commit you made when you committed the submodule changes.

This is a little bit inconvenient, but that's how submodules work.