TL;DR:
Option 1: (Keep using SourceTree):
SourceTree does long thorough refresh. Can be turned off by setting filter to "Modified". (For me I had to set it back and forth once to take effect).

Option 2: (Use Tower/CLI and set showUntrackedFiles)
Option 3: (Use Tower/CLI and With Scalar).
Microsoft has a solution for improving repo performance: Scalar
Long Version:
Had this issue with large repository (has many files). Investigated and found root cause of SourceTree slowness.
You can test issue in command line:
git status -uall
vs
git status -uno
If you find git status -uall
is super slow (same slowness as SourceTree then this is likely your culprit).
Sourcetree always uses -uall
when doing status refreshes.
There is a way to configure your git config file to always skip searching for untracked files when running git status
via:
git config --local status.showUntrackedFiles no
The problem then becomes SourceTree refuses to respect this flag (while other tools like Tower do respect it).
More info can be found here (under status.showUntrackedFiles
section):
https://git-scm.com/docs/git-config
And here (under --untracked-files[=<mode>]
section):
https://git-scm.com/docs/git-status
NOTE: Setting showUntrackedFiles
property to no
on your repo's git config file will mean when you add a new file it won't be picked up by git status
unless you explicitly run it with -uall
param.