After removing the hashbang from my routes using
$locationProvider.html5Mode(true);
Now when visiting a page, for example "domain.com/download", it will work. But if I reloaded this exact page, it would give me an 404 Error. Links like "domain.com/download" can only be opened by typing "domain.com/#!/download". Once it loads it would redirect me to the normal "domain.com/download" page again.
Im using v1.3.2 for routing and v1.6.3 for the rest (dont ask me why I didnt make this site lol).