0

We cannot make the cloud deploy job run the RENDER or DEPLOY in another project's private pool, we can make it use the private pool in the project that hosts the cloud deploy.

Following the documentation of the cloud-deploy setup here: https://cloud.google.com/deploy/docs/execution-environment#changing_from_the_default_pool_to_a_private_pool and here: https://cloud.google.com/build/docs/private-pools/set-up-private-pool-environment#setup-private-connection I have created a clouddeploy.yaml with the following parameters:

apiVersion: deploy.cloud.google.com/v1beta1
kind: Target
metadata:
  name: k8-target
description: apply development
requireApproval: false
gke:
  cluster: projects/development-k8-cluster/locations/europe-west1/clusters/development-k8
executionConfigs:
  - privatePool:
      workerPool: projects/vpchost-project-development/locations/europe-west1/workerPools/cloudddeploy-pool
    usages:
      - RENDER
      - DEPLOY

In summary: there's a cloudbuild project, a k8s project and a clouddeploy project. However, no matter what I do I cannot make the cloud deploy job run the RENDER or DEPLOY in another project's private pool. It does run, but in the clouddeploy project itself. There are no logs, or errors until the deployment phase. Where either cloudbuild starts up inside the clouddeploy project and not in the private pool project or there is an eventual timeout and the pipeline remains stuck as there is no cancel function.

I have given the clouddeploy service account, the cloudbuild service account, a custom service account (not shown in the yaml above) and the default compute service account: owner privileges, cloud deploy runner privileges, cloud build owner and worker pool user privileges.

The request from cloud deploy appears empty except for a run ID that is created when a job is submitted with: cloud beta deploy releases create.

After the deploy release is picked up the job will not do anything until a there is a timeout.

Can anyone see what i've done wrong or has anyone managed to make this work?

EDIT Following a comment from one of the contributors: I expected the 'privatePool' field to be filled and a job running in either the cloud deploy host project or the cloud build project but there is no activity in either.

insidehustle
  • 141
  • 2
  • 7
  • I understood that you have a shared VPC, right? And your Cloud Deploy is set up in another project that your K8S cluster, correct? And I didn't catch your error? Can you share how you deploy? Logs or other stuffs? – guillaume blaquiere Nov 03 '21 at 19:20
  • Thanks for checking, you are right on all points. Cloudbuild is set up in the vpc host project to hook up the service provider network to the shared subnet. There are no logs and no errors. Specifying the resource cloud deploy inside the log browser is unavailable (in our console anyway). Cloud build is empty in the workerpool project and the service accounts (compute/clouddeploy/custom) have not logged any activity in either cloudbuild or cloud deploy project. Have you gotten this to work? That will give me some hope. – insidehustle Nov 03 '21 at 19:37
  • Deploy is done with `gcloud beta deploy releases create placeholder-1 --delivery-pipeline=poc-pipeline --region=europe-west1 --project= testing-clouddeploy-project` – insidehustle Nov 03 '21 at 19:40
  • I never tested this configuration. but why do you want to use a worker pool in another project? If you are in the same VPC, what's the advantage? The cost maybe? – guillaume blaquiere Nov 03 '21 at 20:08
  • Each of the gke clusters requires whitelisting to perform any operations on it and each one is in a different vpc. If I don't use a privatepool and set 0.0.0.0/0 as accepted then everything works. As I understand it: to whitelist cloudbuild/cloud deploy we need to connect each cloudbuild service network to the host project's subnet. – insidehustle Nov 03 '21 at 20:26
  • Hmmm, you want to use Cloud Build private pool, to go through your VPC and to reach a private GKE cluster, correct? And Cloud Build private pool is not able to reach your GKE Cluster, right? – guillaume blaquiere Nov 03 '21 at 20:33
  • Im pretty sure that will work but maybe? I don't know that yet. Cloud deploy can't seem to start the Cloud build in a private pool. It's as if it isn't making the internal api call to the cloudbuild api in another project as far as I can tell now. Which makes me think it's an access thing like the logs not being shown out-of-the-box because the log.viewer access isn't granted. However, I have given the service accounts Owner rights and every other thing related to Cloud deploy in each project. – insidehustle Nov 03 '21 at 20:51
  • By convention, a file containing both the delivery pipeline config and the target configs is called clouddeploy.yaml, and a pipeline config without targets is called delivery-pipeline.yaml. But you give these files any name you want. **Are you maintaining separate files for both the delivery pipeline config and the target configs? or one file?** For more information refer to [delivery configuration file structure](https://cloud.google.com/deploy/docs/config-files#structure_of_a_delivery_pipeline_configuration_file). – Chandra Kiran Pasumarti Nov 05 '21 at 17:50
  • If Google Cloud Deploy is running in a different project from the worker pool's project, make sure the **[service agent](https://cloud.google.com/deploy/docs/cloud-deploy-service-account#service_agent)** has permission on the worker pool in that project. You can use either the **roles/cloudbuild.workerPoolUser** role or the **cloudbuild.workerpools.use** direct permission. – Chandra Kiran Pasumarti Nov 05 '21 at 17:53
  • If you're configuring defaultPool to set the usages (RENDER | DEPLOY), and not an alternate service account or storage location, **include empty braces: defaultPool {}**. For more information refer to **[target definitions](https://cloud.google.com/deploy/docs/config-files#target_definitions)**. – Chandra Kiran Pasumarti Nov 05 '21 at 17:57
  • @ChandraKiranPasumarti we're using one file. it has the roles, i mentioned above that i have given it owner and every other role as well. Yes there's a couple of typos in the file I stripped more away than I should have done while changing it for stackoverflow than I should have done. I'll edit the yaml to remove unnecessary code. – insidehustle Nov 05 '21 at 20:51

1 Answers1

0

You can use another project's private pool, but the Cloud Build instance (and thus where builds show up along with the logs for them) will always been in the project where the Cloud Deploy pipeline lives

Note that in order to make this work, the you will need to grant permission for that pool to be used across projects (See the note under: https://cloud.google.com/deploy/docs/execution-environment#changing_from_the_default_pool_to_a_private_pool)

For logs to show up, you will need to ensure that the service account that is running the build has logging permission in the same project where the delivery pipeline exists.

If the service account being used for the build is not in the same project as Cloud Deploy, you will also need to grant Cloud Deploy act-as permission to use that service account

Edward Thiele
  • 1
  • 1
  • Thank you for contributing Edward. You'll note that I have included that exact link in the question's details. In the comments I have also noted that the permissions were picked up including the logging, owner rights and every other cloud deploy permission. Both to the default service account from the project as well as the cloudbuild account, a custom service account and the cloud deploy service account. I have also enabled the Cloud Deploy, Cloud Build API's. The problem we're seeing is that we cannot start a cloud build in another project. 'You can use another project's private pool', how? – insidehustle Nov 05 '21 at 21:02
  • Have you gotten this to work? – insidehustle Nov 05 '21 at 21:06
  • Hi Luke, I just wanted to clarify on the point of "we cannot start a cloud build in another project", this is correct. The Cloud Build will always occur in the same project as Cloud Deploy, however that build can leverage a pool that lives in another project. – Edward Thiele Nov 09 '21 at 14:58
  • Ok, perhaps the wording needs improvement, I'll edit the question accordingly. I cannot use the private pool of another project in the cloud deploy project. – insidehustle Nov 09 '21 at 22:12
  • You can either use the same Cloud project to create “your private pool and your Cloud Build **builds**” or use **different projects**. If your builds are in a different project from your private pool, set the default project in the gcloud tool to the project where your builds originate. For more information refer to [run builds in the private pool](https://cloud.google.com/build/docs/private-pools/run-builds-in-private-pool) and also refer to the diagram in the [private pool overview](https://cloud.google.com/build/docs/private-pools/private-pools-overview) document. – Chandra Kiran Pasumarti Nov 11 '21 at 20:08
  • @LukeCC, Is your issue resolved or still facing issue? – Chandra Kiran Pasumarti Nov 16 '21 at 11:47
  • @ChandraKiranPasumarti no it is not. I appreciate your suggestions but so far they have not helped. It is not a problem with cloud build, the intention was to build on the private pools we already have. Your last suggestion is unfortunately not relevant, cloud deploy (as far as i can see) does not allow gcloud commands to be issued. Setting the default project does very little as the management layer over cloud build is cloud deploy, not us. I have noticed that some parts of the documentation have been updated and I'm hoping I'll find the root cause/mistake soon. – insidehustle Nov 16 '21 at 23:30
  • Are you using skaffold? Google Cloud Deploy uses [skaffold](https://skaffold.dev/) through Cloud Build to render and deploy your manifests,thus deploying your application. For more info refer to the doc [deploy service architecture](https://cloud.google.com/deploy/docs/architecture). Google Cloud Deploy calls [skaffold render](https://skaffold.dev/docs/workflows/ci-cd/#separation-of-rendering-and-deployment) to render the Kubernetes manifests using this file and [skaffold apply](https://skaffold.dev/docs/workflows/ci-cd/#separation-of-rendering-and-deployment) to deploy them into your target. – Chandra Kiran Pasumarti Nov 23 '21 at 22:33
  • Create a skaffold.yaml configuration file, if you don't already have one, which tells Google Cloud Deploy how to render and deploy your Kubernetes application. You can use [skaffold init](https://skaffold.dev/docs/pipeline-stages/init#deploy-config-initialization) to generate it. For more information refer to the documentation [deploying-application](https://cloud.google.com/deploy/docs/deploying-application) and refer to [Google Cloud Deploy Overview](https://cloud.google.com/deploy/docs/overview#about_skaffold_and). – Chandra Kiran Pasumarti Nov 23 '21 at 22:41
  • @ChandraKiranPasumarti thanks again, yes we're using skaffold. When I generate the configuration with skaffold locally and apply it, it works fine. I looked into that when I faced a different issue. This issue appears to be related to invoking actions between projects. The whole process works if everything is in a single project. I' – insidehustle Nov 29 '21 at 10:21