The thing is that I do not have root permissions on the remote server and can't use the default ssh key location due to the same problem, even if they are for the user I am currently using the destination is OFF LIMITS.
I have found out I can create an ssh in a custom folder now I can't seem to find a way to pass that to git. To make it clear I can't edit the config file nor can I use any root commands.
There might be something like git -i ssh/path
but I can't seem to find any documentation on this issue, for all I know this might not be even possible.
If anyone has found a solution to this any guidance is greatly appreciated!
EDIT : SOLUTION