4

The Google Cloud Function's quota page: https://cloud.google.com/functions/quotas says that the Maximum amount of Data sent by HTTP Function in an HTTP Response is 10 MB. I have files of 100 MB in Google Storage Bucket which I want to stream from the Cloud Function to my application. Due to security aspects of our architecture, my application cannot read directly from the Storage Bucket.

A Sample Go Code in the Cloud Function is:

func MyFn(w http.ResponseWriter, r *http.Request) {

    var appendedBytes []byte
    for {
        decryptedFileBuffer, err := fh.GetNextDecryptedPage()
        if err != nil {
            break
        }
        appendedBytes = append(appendedBytes, decryptedFileBuffer...)
    }
    w.Write(appendedBytes)
}

If I try to get files more than 10 MB using Cloud Function, I get following errors:

Error: incorrect function response. Function invocation was interrupted.

Error: could not handle the request

How can I get files from than 10 MB from the Cloud Function?

user5155835
  • 4,392
  • 4
  • 53
  • 97
  • 4
    Do NOT attempt to stream files from Cloud Functions to users for anything larger than a few dozen/hundred KBs. Use a signed URL so that the user can download directly from Cloud Storage. This saves your billing bandwidth, headaches with Cloud Functions and improves your security. If that is not acceptable, Kolban's answer is another good option to consider. Cloud Storage usually is the correct service to deploy as part of your design. https://cloud.google.com/storage/docs/access-control/signed-urls – John Hanley May 11 '19 at 19:43
  • @JohnHanley Hi. We cannot use the signed url approach since we need to process the file before giving it to the user. So we need an entity (in this case Cloud Function) to process the file and then give it to the user. – user5155835 May 13 '19 at 11:30
  • 1
    Cloud Functions is the wrong service to use. Your use case is not a microservice. You need a compute platform that has network bandwidth. I would still not serve from any compute platform. Read, process and write back to Cloud Storage. Deliver from Cloud Storage. – John Hanley May 13 '19 at 16:06

1 Answers1

4

When we look at the quota page, we see a no against whether or not the quota can be increased. A new player in town is Google Cloud Run ... see here. This provides very similar capabilities to Cloud Functions but uses Kubernetes as the function hosting environment. It's quota allowance is 32MB ... but this too is less than what you need and is also flagged as no for increase.

An immediate thought that occurs to me is to trade convenience (a simple cloud function) for flexibility and run your own compute resources on GCP. For example, running up Compute Engines, App Engines or a Kubernetes cluster where the contained application serves responses through a Web Server is a perfectly fine proposition and not constrained in throughtput. You will have to do the math on billing to determine the cost differentials as the metrics will be different but from a technical perspective, the game changes dramatically. The cloud functions/cloud run capabilities are there primarily to support very short natured microservices and I am guessing that the transmission of 100MB directly is not considered a microservice. It also may not be as cost effective as one may think. It is my understanding that Cloud Functions bills per unit of time utilized. Transmitting 100MBs is likely to be slow latency proposition and may not be as cost effective compared to always running compute as we may think.

Kolban
  • 13,794
  • 3
  • 38
  • 60