2

I have created a web scraping python script and its running fine on my local system it takes 30 mins.

But when I tried to put the script on GCP cloud function it threw timeout after 60004 ms.

2022-03-16T11:41:01.420461007Zget_training_databudock8slftb Function execution took 60004 ms, finished with status: 'timeout'
Function execution took 60004 ms, finished with status: 'timeout'

To complete this I used the following services.

Cloud scheduler -> Pub/Sub -> cloud function

Could you please suggest which GCP service should I pick to run the python script with cost-effective and which runs daily?

vinay kumar
  • 583
  • 6
  • 18

2 Answers2

3

Function execution time is limited by the timeout duration, which you can specify at function deployment time. By default, a function times out after 1 minute or 60000ms which you're experiencing, but you can extend this period up to a maximum of 9 minutes.

You can extend it by editing your deployed function and set the timeout to 540 seconds. enter image description here

For more information, you may also refer on this documentation.

If your scraper took 30mins on your local, then maybe you need to optimize it first and logically set a timer before 9mins and create another schedule by using Google Cloud Client Libraries for google-cloud-scheduler to logically continue where you end and start on it until it finishes the scraping.

Marc Anthony B
  • 3,635
  • 2
  • 4
  • 19
3

You can now use 2nd generation Cloud Functions.

If you create a 2nd gen Cloud Function and use an https trigger, you can set the max timeout to 3600 seconds = 1 hour instead of 10 minutes.

For info on 2nd generation Cloud Functions:
https://cloud.google.com/functions/docs/2nd-gen/overview

See also here on how to combine with Cloud Scheduler:
Do Google Cloud background functions have max timeout?

Sander van den Oord
  • 10,986
  • 5
  • 51
  • 96