3

Scenario-

I am running B* instances on App Engine. I've a background ETL related task(written in python) scheduled as a cron job on App Engine. When time arrives, cron initiates a http request to start the task and runs without returning a response till the task gets completed. When task was executing, it was typically consuming "X" MB of RAM. After the task got finished and returned 200 OK, App Engine Instance monitoring is still showing "X" MB of RAM in use.

Please help me understand the following -

  1. If an instance is running only one task and after completing it, when will memory get freed that was consumed by this task?
  2. Do I need to run gc.collect() to call the garbage collector explicitly to free up RAM ?
  3. The only way to free up RAM is to restart the instance ?

PS: This is not at all related to NDB, my task is taking input from Bigquery, performing some ETL operation and then streaming it to Bigquery.

Ankur Choraywal
  • 162
  • 10

2 Answers2

3

From my observations with an app using lots of memory for StringIO operations:

  • explicitly calling gc.collect() didn't noticeably help (I even suspected for a while that I actually have memory leaks, but it wasn't the case)

  • the memory is not freed after each and every request, but, if the instance remains alive long enough without running out of memory it does eventual appears to be freed now and then. Easy to test - just increase the time between requests to reduce the free memory draining rate. But I couldn't figure out a usable pattern. Note that I observed this only after upgrading to B2 instances, my B1 instances were running out of memory too fast, I never noticed a freeing event with them.

  • using an instance class with more memory (which I tried as a workaround for my instances eventually running out of memory) helped - the memory appeared to be freed more often. It might be because these instances also have a faster CPU (but that's just guesswork).

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • 2nd point that you mentioned seems to work sometimes. I increased the delay between request and I am now seeing memory getting freed now and then. – Ankur Choraywal Aug 06 '17 at 06:26
  • @AnkurChoraywal did you ever find a solution to this, other than increasing the delay between requests. I also wonder if this is specific to App Engine. Would the same behavior be seen on Cloud Run, for example? – BrainPermafrost Jan 05 '20 at 23:02
  • 1
    @BrainPermafrost In my scenario, I solved it by reducing the batch size I query from BigQuery. Processing small batch in the HTTP handler and then re-invoking the same HTTP handler using task queues with query offset. This helped to reduce the memory outage issue by 95% – Ankur Choraywal Feb 05 '20 at 05:51
2

There are a few questions on StackOverflow describing similar memory issues for tasks when using ndb on app engine. Here is one example.

The issue is that app engine doesn't clear the ndb context cache upon the conclusion of a task so context cache continues to hog your memory long after the task completes.

The solution is to not use or clear the context cache during your tasks. Here are a few ways:

  • Bypass caching with key.get(use_cache=False)
  • Call ndb.get_context().clear_cache() at appropriate times.
  • Disable caching for all entities of a kind by adding _use_cache = False to your model definition.
new name
  • 15,861
  • 19
  • 68
  • 114
  • There's an extensive discussion in this [bug report](https://issuetracker.google.com/issues/35895760). – snakecharmerb Jul 11 '17 at 15:03
  • Thanks Jeff ! But this is not at all related to NDB. I am running a task that takes input data from BIGQUERY, perform some ETL operation to this data and then stream it to BIGQUERY and then task get finished. Can you please tell me how we can make memory free after finishing the task ? – Ankur Choraywal Jul 12 '17 at 07:53