I hope this helps some people as it took me some time to figure it out
I kept trying with VS code git plugin for a while but then I tried the following:
1- Open a terminal & cd to your project directory
2- Run the following command:
git clone put-your-ssh-git-repo-url
3- Then I was prompted to accept that authenticty of the host couldn't be verified & after accepting that it started getting the repo
So I believe that vs code didn't clone because it couldn't get my approval for this authenticity step (step number 3) as it was trying to authenticate with gitlab implicitly, I think in any other project on the same machine it shall work as now this pc has: Permanently added the host (in my case gitlab) (ECDSA) to the list of known hosts.
Note:
As per this excellent answer here , you can add the host for your git project to the known (authorized) hosts by running the following in the terminal (replace github.com with your host, in my case gitlab.com for example)
ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts
After doing this step above, cloning directly from vs code git plugin shall work