GCP Cloud Function - ERROR fetching storage source during build/deploy

Question

Running into problems building deploying functions. When trying to programmatically deploy the function I get the following output in builder logs (ERRORS).


2020-10-20T02:22:12.155866856Z starting build "1fc13f51-28b6-4052-9a79-d5d0bef9ed5c"
 I 
2020-10-20T02:22:12.156015831Z FETCHSOURCE I 
2020-10-20T02:22:12.156031384Z Fetching storage object: gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655 I 
2020-10-20T02:22:16.698838385Z Copying gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655... I 
2020-10-20T02:22:16.848908949Z / [0 files][    0.0 B/  7.6 KiB]                                                
/ [1 files][  7.6 KiB/  7.6 KiB]                                                
 I 
2020-10-20T02:22:16.848965618Z Operation completed over 1 objects/7.6 KiB.                                       I 
2020-10-20T02:22:17.848654342Z replace /workspace/pb/__init__.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL I 
2020-10-20T02:22:17.848694354Z (EOF or read error, treating as "[N]one" ...) I 
2020-10-20T02:22:17.848697209Z Archive:  /tmp/source-archive.zip I 
2020-10-20T02:22:17.848698273Z   inflating: /workspace/main.py       I 
2020-10-20T02:22:17.848699274Z    creating: /workspace/pb/ I 
2020-10-20T02:22:17.848700378Z   inflating: /workspace/pb/__init__.py   I 
2020-10-20T02:22:17.848701376Z   inflating: /workspace/pb/${FILENAME}.py   I 
2020-10-20T02:22:17.848702578Z   inflating: /workspace/requirements.txt   I 
2020-10-20T02:22:18.477862579Z Fetching storage object: gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655 I 
2020-10-20T02:22:21.795484700Z Copying gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655... I 
2020-10-20T02:22:21.937040057Z / [0 files][    0.0 B/  7.6 KiB]                                                
/ [1 files][  7.6 KiB/  7.6 KiB]                                                
 I 
2020-10-20T02:22:21.937094331Z Operation completed over 1 objects/7.6 KiB.                                       I 
2020-10-20T02:22:22.934311112Z replace /workspace/main.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL I 
2020-10-20T02:22:22.934315945Z (EOF or read error, treating as "[N]one" ...) I 
2020-10-20T02:22:22.934318352Z Archive:  /tmp/source-archive.zip I 
2020-10-20T02:22:23.569955715Z Fetching storage object: gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655 I 
2020-10-20T02:22:26.802351633Z Copying gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655... I 
2020-10-20T02:22:26.867398009Z / [0 files][    0.0 B/  7.6 KiB]                                                
/ [1 files][  7.6 KiB/  7.6 KiB]                                                
 I 
2020-10-20T02:22:26.867408400Z Operation completed over 1 objects/7.6 KiB.                                       I 
2020-10-20T02:22:27.863532350Z replace /workspace/main.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL I 
2020-10-20T02:22:27.863536996Z (EOF or read error, treating as "[N]one" ...) I 
2020-10-20T02:22:27.863538973Z Archive:  /tmp/source-archive.zip I 
2020-10-20T02:22:27.995306558Z ERROR I 
2020-10-20T02:22:27.995310537Z ERROR: error fetching storage source: generic::unknown: retry budget exhausted (3 attempts): fetching gcs source: unpacking source from gcs: source fetch container exited with non-zero status: 1 I

Confusingly, if I deploy the function from the UI, it builds/deploys fine. That makes me think this is some sort of IAM/permissions around the builder/agent. But, am rather lost. Any tips/thoughts? Thanks!

Hi @bruce indeed, seems to be a permission issue, considering that via UI you are able to. When deploying via command, it's usually the service account the agent that will be building and deploying the application. Considering that, could you please take a look at this documentation [here](https://cloud.google.com/functions/docs/concepts/iam) and get more details on service accounts? This way, you will be able to check those that are being used when you are deploying your application and then, confirm if they have the same permissions as your account. — gso_gabriel, Oct 20 '20 at 13:16
Thanks, @gso_gabriel -- There is indeed something special with the service accounts. And, I'm deploying everything else via terraform. I had worked around via just deploying the specific resources via `gcloud` command in extra deploy steps... But, yes, I should take the time to sort out the permissions. — bruce, Nov 18 '20 at 19:23
It has been awhile -- but, IIRC (from many months back), it was the permissions for using Cloud Build to be able to build/deploy the function. — bruce, Feb 16 '21 at 07:33

score 27 · Answer 1 · edited Feb 09 '21 at 08:03

27

Had the same error message after spending a day troubleshooting I finally found a way.

I noticed I had node@15 installed on my PC and node@12 for firebase function.

All I need do was to

install node@12 on my PC
unlink the previous node (i.e node@15) and
link to the newly installed node@12.

Boom! it uploaded.

edited Feb 09 '21 at 08:03

coderina

1,583
13
22

answered Jan 18 '21 at 18:24

Prodigy

2,094
24
30

4

This answer should be accepted. I changed my Node@15 to Node@12. Unlink node15, link node12 then it works perfectly. – Harry T. Jan 19 '21 at 09:16
@TruongHieu I'm glad it helped :-) – Prodigy Jan 19 '21 at 16:48
I want to add that I use Firebase functions, and it worked for me also. Yeah!!! Thank you :) – João Melo Jan 30 '21 at 15:41
so you can downgrade your version with `n` with LTS version `$ npm install -g n $ n lts` – Julien B Feb 04 '21 at 21:51
Amazing, I never would have figured that out myself! – user3413723 Feb 08 '21 at 17:53
I really wonder how this cloud builder error can be the result of the local NodeJs version. If the firebase / gcloud CLI has this kind of dependency they should really enforce the Node version in the CLI. – Thijs Koerselman Feb 09 '21 at 12:02

score 6 · Answer 2 · answered Jan 31 '21 at 14:24

I had the same problem deploying cloud functions on my newly started Firebase projects. My local version of node was 15

For beginners (like me) which were struggling downgrading node on Mac, you can use the following commands:

sudo npm install -g n
sudo n 12

Simon · Answer 3 · 2021-01-21T21:06:50.860

4

As Prodigy mentioned, something must has broken the Cloud Functions deployment in the latest Node v15.6.0 (or another recent one).

Has the same issue today with firebase deploy. Resolved by rolling back to Node 12 (and I am sure any other more recent version works great).

edited Jan 21 '21 at 21:06

answered Jan 19 '21 at 00:14

Simon

81
6

2

I had exactelly the same issue and your solution helped me. Thanks – Samuel L Jan 21 '21 at 17:30

score 2 · Answer 4 · answered Jan 21 '21 at 20:21

2

I can confirm the issue came with node update to 15.6.0. Running Manjaro, updated my OS, rolling back to node 12 (with nvm) solved the issue!

answered Jan 21 '21 at 20:21

23Ro

33
7

Likewise. I had this failing on a Gitlab build which was using `node:latest` as the container. Turned out it was 15.6.0. I moved down to `node:10.23.1` and it worked. – worldofchris Jan 27 '21 at 12:24

score 2 · Answer 5 · answered Jan 23 '21 at 04:52

2

Even downgrade nodejs to version 15.5.1 works for me. GC function definitely don't like version 15.6.0

answered Jan 23 '21 at 04:52

Dennis Liger

1,488
2
13
28

2

this was the solution for me!!! I ended up removing node via homebrew with `brew unlink node` and `brew uninstall node` and then installed `n` via homebrew `brew install n` and then install v15.6.0 with `n 15.6.0` – mesqueeb Feb 02 '21 at 03:33

score 0 · Answer 6 · answered Nov 29 '20 at 19:18

0

It has nothing to do with permissions. It must be something else. Maybe google sets limits as to how big cloud function should be.

answered Nov 29 '20 at 19:18

Donatas Svilpa

291
2
3

score 0 · Answer 7 · answered Jan 21 '21 at 02:49

0

I had a similar issue today. It turned out that on one function I was using a timeoutSeconds that exceeded the max of 540 as specified here.

answered Jan 21 '21 at 02:49

Ryan Biwer

324
4
14

score 0 · Answer 8 · answered Mar 04 '23 at 17:46

0

Update for 3/2022, firebase deploy no longer supports node@12. I had node@18 installed, and by setting

"engines": {
    "node": "18"
  }

in package.json I was able to fix this issue

answered Mar 04 '23 at 17:46

app_sciences

783
1
7
14

GCP Cloud Function - ERROR fetching storage source during build/deploy

8 Answers8

Linked