3

I am trying to set up a monorepo with multiple cloud functions written in Python. I am currently using Cloud Build and structure like this:

.
├── deployment
│   └── cloudbuild.yaml
├── main.py
└── requirements.txt

which with this Cloud Build YAML code deploys well:

steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args: [
      'functions', 'deploy', '$_FUNCTION_NAME',
      '--trigger-resource', '$_TRIGGER_RESOURCE',
      '--trigger-event', '$_TRIGGER_EVENT',
      '--runtime', 'python37',
      '--memory', '1024MB',
      '--region', 'europe-west1'
    ]

Now my intent is to move towards this structure:

.
├── first_function
│   ├── main.py
│   └── requirements.txt
├── second_function
│   ├── main.py
│   └── requirements.txt
└── cloudbuild.yaml

With triggers set up to watch changes within the respective subfolders, injecting the function name as env variable and deploying the correct function. This is the TF idea of setup:

resource "google_cloudbuild_trigger" "first_function_trigger" {
  project = google_project.my_project.name
  name = "trigger-first-function"
  description = "Trigger for deploying first function"

  trigger_template {
    repo_name = google_sourcerepo_repository.functions.name
    branch_name = "master"
    dir = "first_function/**"
  }

  substitutions = {
    _TRIGGER_RESOURCE = google_storage_bucket.my_bucket.name
    _TRIGGER_EVENT = "google.storage.object.finalize"
    _FUNCTION_NAME = "first_function"
  }

  filename = "cloudbuild.yaml"
}

However, here is the catch:

All arrangements, with specifying --source in the gcloud functions deploy command, just keep giving me errors such as:

ERROR: (gcloud.functions.deploy) argument --source: Provided directory does not exist

This error occurrs when I try these values:

1. --source=.
2. --source=./first_function
3. --source=./first_function/

Number three works locally when gcloud functions deploy is called from the root folder. I read about the approach of specifying the repository in GCP - but that's an extra data loading operation, no? The source code is already there - this is a trigger for changes in the repository.

When no --source is defined, this is the error I get:

ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Build failed: Build error details not available

I know that Cloud Build is a fairly young service and changes very rapidly, but is there a way now to arrange the folders or set up the cloud build YAML so that the functions get deployed correctly? I really don't want to create a separate repository for every single 100-line function.

Michal
  • 15,429
  • 10
  • 73
  • 104
  • What are you passing as `--source`? – Dustin Ingram Mar 04 '20 at 22:08
  • @DustinIngram you're right, I forgot to put that in - thanks for the heads up. I edited the question to provide that information now. – Michal Mar 05 '20 at 08:15
  • Why do you use terraform for triggering the CloudBuild? Why do you not trigger directly Cloud Build? or why do you not deploy your function with Terraform? I missed something. – guillaume blaquiere Mar 05 '20 at 09:54
  • @guillaumeblaquiere TF is used for configuring all other infra (VPC, GKE, Vms etc). The config for Cloud Build is included as a CI/CD module. The function is deployed using the cloudbuild.yaml. I believe this is not an important detail to the actual problem of deploying multiple functions from a monorepo, but if you think it is, I'm happy to listen. – Michal Mar 05 '20 at 10:31
  • @guillaumeblaquiere sorry I might have misunderstood - the TF simply only defines the Cloud Build trigger, it does not trigger it. Cloud Build is triggered automatically on code push. It is the same as if I defined it via `gcloud` or the UI console. – Michal Mar 05 '20 at 10:33
  • @Michal how did you mange to watch changes and only deploy commited functions? – czmarc Jun 24 '20 at 18:45
  • @czmarc here's pastebin of [shortened TF config](https://pastebin.com/D8i6CW1Q) with the config for Cloud Build. The trick is to separate your code into **common folder**, **function specific folder** and have **main.py** and **requirements.txt** in root. The **main.py** file here then serves as the main router for all other functions. The watch is not flawless (you get false positives with changes in requirements.txt or common), but good enough. – Michal Jun 25 '20 at 08:08

1 Answers1

4

I was unable to reproduce your issue with just Cloud Functions + Cloud Build. With the following structure:

.
├── cloudbuild.yaml
├── first_function
│   ├── main.py
│   └── requirements.txt
└── second_function
    ├── main.py
    └── requirements.txt

And the following cloudbuild.yaml:

steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args: [
      'functions', 'deploy', 'first_function',
      '--trigger-http',
      '--runtime', 'python37',
      '--region', 'us-central1',
      '--source', 'first_function'
    ]
  - name: 'gcr.io/cloud-builders/gcloud'
    args: [
      'functions', 'deploy', 'second_function',
      '--trigger-http',
      '--runtime', 'python37',
      '--region', 'us-central1',
      '--source', 'second_function'
    ]

I was able to deploy both functions.

Is it possible the source flag is not being set correctly?

Dustin Ingram
  • 20,502
  • 7
  • 59
  • 82
  • It might sound crazy. But I couldn't get your code to work. And then...I put `--source` as the last parameter and it worked. Would you mind trying putting `--source` before `--trigger-http` to see if you get the same error? Or did I miss something in the documentation that clearly states that it won't work if the order is different? – Michal Mar 06 '20 at 14:16
  • 2
    The order shouldn't matter. Changing the order didn't cause it to fail for me. – Dustin Ingram Mar 06 '20 at 19:58
  • Interesting. Well thank you for your assistance - I flipped the order again and it did fail for me, so should it be of interest to see why (for internal GCP purposes), I'm happy to assist with logs etc - I'm on the official GCP Slack, so I'm happy to follow up on there. I'm marking your answer as accepted to close this up here! – Michal Mar 07 '20 at 09:56
  • The only thing I can think of, maybe your source string has a newline character in it? – Dustin Ingram Mar 07 '20 at 14:42
  • I doubt it - I pass it as `$_FUNCTION_NAME` which is a variable in Cloud Build. As you can see in my TF file, it's passed as `_FUNCTION_NAME = "first_function"`. Doesn't seem like a new line there. Unless, of course, terraform adds it (I guess unlikely?). – Michal Mar 07 '20 at 17:12
  • This might be a terraform issue, can you try to build without using terraform and check if the issue persists? – Emil Gi Mar 10 '20 at 13:46