Branch names, like all refs in Git, are case sensitive. They are often stored in the file system, which means that on some systems, it is not generally possible to store two names differing in case (by whatever means that file system determines that). However, they remain case sensitive, and Git will not consider two names differing in case to be identical.
Note that when references are packed into the packed-refs
file, they are not stored with their names as file names, so every attempt to access them in a case-insensitive way will fail. There is a future design for references, called reftable, that will never store names in the file system and thus with that design references can never be treated case insensitively.
I should also point out that references are also not guaranteed to be in UTF-8, since they can contain bytes not valid in UTF-8. (However, if you choose non-UTF-8 names, you're in for a world of pain.) Even if you assumed they were in Unicode, in general it is impossible to properly case-fold Unicode text in a locale-insensitive way, so Git doesn't try to do that.
If you try to treat references in a case-insensitive way, it will break, as you've seen, so your best bet is to avoid that.