2

I have IAP configured and use it to connect to several instances in GCP. All the pre-requisite permissions and firewall rules I should need are in place (IAP-secured Tunnel User, owner, security admin) and this works flawlessly on all other instances.

However I recently added a Ubuntu 22.04 LTS instance into the mix and cannot connect to it via IAP. I can use the ssh option in GCP to connect but not IAP. When attempting to connect via IAP I have received the errors below:

enter image description here enter image description here

When I attempt to connect via SSH from another GCP instance in the same network I get:

enter image description here

Due to these errors I suspect it must be something to do with the keys, but really not sure what else to check.

Additionally, I created a new test instance running Ubuntu 20.04 and IAP connects fine just as it does on all my other instances, It just doesn't connect if the instance is running Ubuntu 22.04.

Any thoughts?

Edited to add screenshot of ssh test from another system on the same network. SSH Test was done with a locally created account and my GCP credentials. enter image description here

Ricky
  • 61
  • 7
  • Did you enable a firewall rule to allow IAP traffic into the instance? Allow traffic from `130.211.0.0/22` and `35.191.0.0/16` on TCP port 80. – John Hanley Aug 25 '22 at 22:35
  • John, thanks for the suggestion. I initially followed the steps here: https://cloud.google.com/iap/docs/using-tcp-forwarding but the ip range that I used was 35.235.240.0/20 (I don't have any load balancing configured) . However the existing configuration continues to work for all existing and test new instances that I have created thus far, EXCEPT for any built using Ubuntu 22.04 (Ubuntu 20.02 works fine) I'll do some research on those other IP's. Thanks for the suggestion! – Ricky Aug 25 '22 at 22:57
  • I assumed that you have the firewall rule, but you must assign the rule to the instance. Double-check the VM's **network tags**. – John Hanley Aug 25 '22 at 23:26
  • I have IAP fw rule set for all instances (Including any new ones) created in my network. I double checked the other tags and everything looks correct. It's strange that I can create a new instance of any type (windows, debian, centos, and even Ubuntu so long as the version is 20.04 or earlier) and IAP connectivity works as expected on all of them,, but not when creating a new Ubuntu 22.04 vm. Is it possible that Google didn't add a necessary key or permission when this image was added to GCP? or that Ubuntu 22.04 has changed in some way to cause this connectivity to be broken? – Ricky Aug 29 '22 at 14:09
  • 1
    Do your VMs have public IP addresses? If yes, use a regular ssh tool to connect. You can then verify SSH access versus another potential issue. – John Hanley Aug 29 '22 at 19:59
  • The VM does have a public IP address, but I have port 22 disabled for anything external of my subnet, but from the internal range I have tested and can successfully ssh to the instance, but ONLY when I use a local test account that I created. My GCP account credentials (IAM admin/IAP admin) are not accepted. I will add a screenshot of this to the original post. – Ricky Aug 30 '22 at 21:16
  • 1
    I recently found out that RSA generated ssh keys are not longer considered safe by ubuntu 22.04's ssh client, and I had a similar problem not having permission to access bitbucket even though the rsa keys were in place. Eventually I looked for differences between ubuntu 20.04, what I had before, and the new 22.04 and came across a comment stating that the solution is to generate ecdsa keys instead. To my surprise that's the only thing that worked. I am now having a similar problem trying to access gcloud which fails to connect precisely after generating the public/private ssh rsa keys. – Gabriel Em Sep 01 '22 at 19:33
  • @GabrielEm - RSA keypairs are fine. You must have misread something. Or you read something about SHA-1 or smaller bit numbers. – John Hanley Sep 01 '22 at 19:53
  • 1
    @Ricky I tried to recreate your issue, but it is not happening to me. I instantiated 2 Ubuntu VMs in the same VPC network, one with 20.04 LTS OS and the second with 22.04 LTS OS. And using [gcloud auth login](https://cloud.google.com/sdk/gcloud/reference/auth/login), I can SSH using IAP from first to second without issues. Have you tried clearing the known_hosts file? – Gabriel Robledo Ahumada Sep 02 '22 at 20:01
  • @GabrielRobledoAhumada You may be onto something with the known_hosts file. When I run " gcloud compute ssh --verbosity=debug, I connect and the prompt shows the server name in the prompt. But when I run the same command with the new server name it shows the IP rather than the name. On the old server I checked the path /home//.ssh/known_hosts and do see the file with expected contents. Yet, on the new server the known_hosts file is missing. So it's not getting created. – Ricky Sep 08 '22 at 15:17
  • @GabrielRobledoAhumada Also, ssh'ing from the old server to the new with "gcloud compute ssh --project= --zone= --tunnel-through-iap" the results show "ERROR: (gcloud.compute.ssh) Could not fetch resource: - Request had insufficient authentication scopes". But I get the same results, from new to old. But it connects find from gcp cloud shell. Yet, IAP Desktop has no issues allowing me to connect to the older server, (just not the newer Ubuntu one) so this may be a red herring. I appreciate your help! it's given me some direction to troubleshoot. – Ricky Sep 08 '22 at 15:24
  • @GabrielEm - The SSH key type **ssh-rsa** was dropped in OpenSSH version 8.8. That key type is RSA with SHA-1 hashing. The replacement is **rsa-sha2-256** which is RSA with SHA-2 hashing. The RSA key itself has not changed. It is the hashing algorithm (SHA-1) used by the SSH protocol that has been deprecated. In summary, the RSA key is not affected, only the algorithm that SSH uses has been modified. – John Hanley Sep 21 '22 at 18:19
  • @JohnHanley Yeah, that's correct. Nonetheless this change started causing problems because of differences between the OpenSSH clients/servers (for instance keys added to bitbucket servers managed by many work places using bitbucket were no longer working with clients on Ubuntu 22.04, until everyone caught up on the changes and either updated the ssh clients everywhere or specified any algorithm that works, including rsa-sha2-256). I wrote my original comment just in case there's a chance for this fairly recent issue to play a role in OP's issue. – Gabriel Em Sep 24 '22 at 14:23

2 Answers2

1

A number of things can be happening here, but a good troubleshooting point would be to take a look at the local stored SSH host keys. From what I have seen, all your OS are Linux VMs, so you can start by deleting the known_hosts file and trying to connect again.

Gabriel Robledo Ahumada
  • 1,542
  • 2
  • 14
  • Thanks for your input, I have a mix of windows, centos, & Ubuntu. The only instance{s} with which I have connectivity issues is Ubuntu 22.04 LTS, All existing instances connect just not a new ubuntu 22.04 (I can make new centos an they will connect) I took a look at the "problem instance "and after connection attempts and nothing is in the local SSH host key. For further clarification I am attempting to connect using IAP Desktop application. – Ricky Sep 19 '22 at 16:54
  • @Ricky I posted the answer based on the previous comments, if the answer was helpful please consider upvoting or accepting it. – Gabriel Robledo Ahumada Sep 19 '22 at 17:07
  • I'd love to answer as completed, but i'm still having problems. However after going through all the previous troubleshooting steps again, I have found that I am able to use cloud shell to ssh successfully as well as use the ssh option to connect from the vm instance list in compute engine. It looks like the issue may be something with the IAP Desktop application that I am using. – Ricky Sep 20 '22 at 19:31
  • As stated previously, the issue may actually be with the IAP Desktop application. I believed this to be a GCP product, but that now appears to be incorrect. So the issue that I have connecting with Ubuntu 22.04 may well prove to be an issue with that product and not via the other methods tested within GCP. I will follow up with that vendor to see what I can learn. I greatly appreciate the troubleshooting assistance, it has been very helpful! – Ricky Sep 20 '22 at 19:33
1

I have finally found the resolution to this issue. Ordinarily I believe the troubleshooting steps posted by @GabrielRobledoAhumada and others would likely resolve, but my issue turned out to be with the use of IAP Desktop, an application that I errantly believed to be a Google created/supported application. Once this was realized I reached out the the vendor who worked with me to resolve.
The issue turned out to be that by default, "Ubuntu 22.04 does not allow RSA for SSH public key authentication anymore" a change that was implemented a few releases ago. The IAP Desktop application I was using was still trying to use RSA as the default connection setting, Once I changed the 'key type' in the IAP Desktop app to EDCDA NIST P-256 I was able to use it to connect successfully. Older version of Ubuntu and other OS's still allow the RSA keytype and is thus the reason that I could connect to basically any OS type other than Ubuntu 22.04. Thanks to those that offered assistance, and my apologies that I was under the false impression that I was using a GCP tool which made legitimate troubleshooting difficult.

Ricky
  • 61
  • 7
  • @GabrielEm After working with the vendor to find a resolution, it turns out that your answer was on the right track in finding the solution. As such, I am upvoting your previous response! Thanks for weighing in! – Ricky Sep 21 '22 at 17:51