3

Our project is using a Serverless VPC access connector to allow access to DB over private IP from cloud functions and cloud runs. It was working flawlessly for a few months, but today I tried to deploy one of the functions that use such a connector and I got the message:

VPC connector projects/xxxx/locations/us-central1/connectors/vpc-connector is not ready yet or does not exist. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.

enter image description here

I went to the Serverless VPC access view and found out that indeed the connector has a red marking on it. When I hover on it it says

Connector is in a bad state, manual deletion recommended

but I don't know for what reason, Link to logs doesn't show anything for the past 3 months.

enter image description here

I tried to google about the such error but without success. I also tried to search through logs but also didn't find anything relevant.

I'm looking for any hints:

  • Why it happened?
  • How to fix it? I don't want to recreate the connector, it is related to many functions, and cloud runs
mjpolak
  • 721
  • 6
  • 24

1 Answers1

3

As the issue was blocking us from the deployment of cloud functions I was forced to recreate the connector.

But this time API returned an error:

Error: Error waiting to create Connector: Error waiting for Creating Connector: Error code 7, message: Operation failed: Google APIs Service Agent (<PROJECT_NUMBER>@cloudservices.gserviceaccount.com) needs editor role in the project.

After adding such permission old connector started to work again...

Before there was no such requirement, but it changed in meantime.

Spooky, one time something works other not.

mjpolak
  • 721
  • 6
  • 24
  • 1
    I am having the same problem. I don't trust myself to recreate the connector since if I do it wrong, I'll be bringing down our live site. I tried adding editor permissions to any related service account and was unsuccessful. Will ask my GCP consultant. I agree GCP has stuff working one day and completely broke the next, it's happened so many times to our prod site it's kinda a scary reality. – Kevin Danikowski Dec 08 '22 at 16:08
  • 1
    Turns out it got solved on its own. Google had some resource issues and I think that caused it. After waiting 24 hours it fixed itself. – Kevin Danikowski Dec 09 '22 at 15:35
  • I've encountered the same problem while tearing down my infra using pulumi (similar to terraform). I also saw the "VPC access connector is in a bad shape" and the API also returns for me the same error about adding the editor role. When I go the Google Cloud Console UI to delete the connector is also fails to delete with the same error as the API ... which was really unexpected as I'm logged in with my personal account that has the Owner role.... – James Lawson Jan 08 '23 at 14:50
  • @KevinDanikowski thanks for the info, really valuable feedback. It's good to know that I'm not doing anything wrong on my side. Did the GCP consultant recommend any next steps from Google or provide any advice or workarounds? If more people encounter this on StackOverflow maybe it's worth raising a bug with GCP (if one hasn't been raised already?). – James Lawson Jan 08 '23 at 14:58
  • 1
    @JamesLawson Honestly i've found that lots bugs like this with GCP. I don't trust them for reliability. I've had load balancers fail for no reason a few times, had to recreate it to fix it. This one wasn't easy to recreate so I'm glad it fixed itself. i think the only solution is to rebuild it tho. Otherwise, just wait. in my case, it didn't stop working, we just couldn't connect to launch new cloudrun containers. so we were able to wait it out – Kevin Danikowski Jan 08 '23 at 22:04