There isn't a way to do OS configuration in Git configuration. However, there are two possible approaches.
One is to to take advantage of the fact that there are two per-user config files, ~/.gitconfig
and ~/.config/git/config
(although the latter may differ if you have $XDG_CONFIG_HOME
set). I store my common configurations in the former, and any custom, per-machine configuration (which is not managed by Git) in the latter, such as credential helpers and signing subkeys. It is possible to have a script in your dotfiles that copies an appropriate file into the latter if you do manage it with Git.
The other option is to specify the credential helper as a shell command, which of course has all the power of the shell. So you could do something like this:
[credential]
helper = "!f() { if [ \"$(uname -s)\" = Linux ]; then git credential-cache --timeout 3600 \"$@\"; else git credential-osxkeychain \"$@\"; fi; };f"
That will invoke the proper helper depending on the OS you're running on.