38

I try to deploy application on the GC AppEngine. There are no errors during deploy process but application doesn't work (Just show loading page). The only one strange raw in logs 

OpenBLAS WARNING - could not determine the L2 cache size on this system

By the way - it works well on my local machine.
This is python web app based on Dash framework

My app.yaml:

runtime: python37
service: service-name
instance_class: F2

entrypoint: gunicorn -b :$PORT main:app.server

Requirements.txt:

Flask==1.0.2
dash==0.34.0
dash-html-components==0.13.4
dash-core-components==0.41.0
dash-table==3.1.11
gunicorn==19.9.0
google-cloud-pubsub==0.37.2
requests==2.21.0
pandas==0.23.4
Pizza eu
  • 1,419
  • 1
  • 14
  • 27
  • 1
    OpenBLAS is platform dependency C library. I don't think app-engine will support it. – howie Mar 06 '19 at 08:50
  • 1
    Remember you are running on a virtualized environment, not a physical computer or processor. Libraries like this are not as effective in VMs. The reason for the warning is that the VM does not have access to the CPU cache. App Engine is a virtualized environment within another virtualized environment. – John Hanley Mar 06 '19 at 10:07
  • Is there anyway to disable this warning? – yudhiesh Nov 01 '22 at 03:02

4 Answers4

40

I just had your same problem with pandas and Dash and found your question (hoping it would give me some light). After being stuck for several hours, I found the answer, and came back to share :-)

If the only error that you're seeing is the OpenBLAS warning, most likely the app is working well. After debugging this problem for several hours, I found that as Dash and Pandas consume a lot of memory, the F2 instance is not able to handle the web app properly and fails due to lack of RAM memory. Please try changing in your YAML/JSON configuration file your instance to the highest possible automatic unit with more RAM memory, and then it will probably work:

instance_class: F4_HIGHMEM

EDIT: Google App Engine now supports more instance types. Check the docs of instance types: standard instances

Instance types

In addition, please keep in mind that the first time you run this web app, it will take considerably more time to execute. If you check the logs you'll have several prompts like the one below. Just wait a little bit more

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

David Olmo Pérez
  • 745
  • 7
  • 14
  • This is exactly the answer I was looking for. However, it still takes about 15 seconds to spin up my Dash + Flask application, which is a bit annoying. For heavier applications I would recommend leaving 1 instance on all the time. But watch out that this could be relatively costly! – Cloudkollektiv Jul 09 '19 at 14:12
  • 4
    I tried having 1 instance running all the time and didn't notice much difference in terms of startup. Google Cloud has implemented some sort of standby / warm start which works really great in my case. BUT, if you don't have any request for more than 2 hours, then you'll face those 15 secs again. I finally scripted a google cloud function (similar to AWS lambda) that is executed every 2 hours and makes a dummy request to the server, so it doesn't go off. This has worked very well in my case as it is cheap (this is still in the free zone) – David Olmo Pérez Jul 23 '19 at 23:53
7

I personally solved it by adding a timeout to gunicorn, as the default timeout is only 30 sec

entrypoint: gunicorn -b :$PORT main:app.server --timeout 120

I found this solution as I tried the following:

  • switched from a F1 to a F4_1G instance: still had the same warning
  • switched from App Engine Standard to App Engine Flexible environment (which I highly DO NOT recommend, as app engine flexible instances not properly shut down (deleted) can cost you a lot of money: please see here for reference Pricing of Google App Engine Flexible env, a $500 lesson), with 16gb of ram and 4 CPU -> eventually got a different warning "[CRITICAL] WORKER TIMEOUT" which pointed me to this post: Gunicorn worker timeout error , which is where I found this solution.

Now my app works well, even with a F1 instance

tim
  • 103
  • 2
  • 6
2

I experienced the same error log using a AWS Lambda function. My function was processing a decent number of records so I needed to increase two settings in my lambda function configuration:

  • Increase the size of Memory.
  • Increase the Timeout in seconds.

Hope that helps.

yaach
  • 386
  • 2
  • 3
1

Don't forget to change the number of workers ! there is a default number of workers in AppEngine instances, so your application is multiplied by this number on the same instance

https://cloud.google.com/appengine/docs/standard/python3/runtime

enter image description here

Arrajj
  • 167
  • 2
  • 12