This is only happening for me with my Java projects. I'm using Intellij but is the same with netbeans. These are the steps I've taken to remake this problem in a simple way. The folder structure I'm using is because of the requirements of my class.
Make clean github repo
mkdir classwork with a README.md
mkdir summatives with a README.md
create branch-one
create branch-two
push both to remote
checkout branch-two
Make a HelloWorld Maven project in classwork folder
add commit and push to branch-two
checkout branch-one
Now after switching to branch-one which should only have two empty folders there will still be my hello world project in the classwork folder. The project will be inaccessible sometimes being empty or with a few random files. Also looking on github branch-one is empty as expected.