As far as Git is concerned, all branch and tag names are case sensitive by design, so while it may happen that in some cases you can access a branch case insensitively, you can't in all cases, so it's better to pick a convention and stick with it (which usually means lowercase).
This is because Git often, but not always, stores branches and tags in the file system. When it does this, it's possible for those branches and tags to be accessed in a case-insensitive way if the file system on your computer is also case sensitive. Therefore, on a Linux system, these names are always case sensitive, and on macOS or Windows, they sometimes are and sometimes not.
When you pack a repository, however, Git packs all of the branch and tag names into a file and removes them as files. In that case, all of those branch and tag names are case sensitive, even on a case-insensitive file system.
Similarly, you can often use the special reference name HEAD case insensitively on an appropriate file system, but this will fail in various cases, so it's not recommended.
The reason that Git doesn't permit case-insensitive behavior is because the encoding of reference names need not be UTF-8, and case folding in general is a hard problem, so it's not possible to sanely produce a good case-insensitive behavior.