0

I have a Google App Engine app running (Flask app) that seems to have a memory leak. See the plot of the memory usage below. The memory usage continually creeps up until it hits the limit and the instance is shutdown and a new one is started up.

It's a simple API with about 8 endpoints. None of them handle large amounts of data.

I added an endpoint that takes a memory snapshot with the tracemalloc package, and compares it to the last snapshot and then writes the output to Google Cloud Storage.

I don't see anything in the reports that indicates a memory leak. The peak memory usage is recorded as about 0.12 GiB.

I am also calling gc.collect() at the end of every function that is called by each endpoint.

Any ideas on how to diagnose this, or what might be causing it?

enter image description here

Pablo Almécija Rodríguez
  • 1,470
  • 8
  • 14
BrainPermafrost
  • 644
  • 2
  • 7
  • 20
  • There are quite a few similar questions here, like [this one](https://stackoverflow.com/questions/56439727/python-google-app-engine-can-not-release-memory-of-object-array). Unfortunately, I've never seen a good answer. It seems that memory should be recycled more quickly. – new name Sep 20 '19 at 00:28
  • @BrainPermafrost - have you been able to solve this, please? If yes, I would appreciate any info on how. – KaaRoy Oct 07 '20 at 19:41
  • I am currently experiencing the exact same problem. It was happening so often on my F1 instance that I finally upgraded to F2... Within 24 hours it's happened again at the higher 768 MiB threshold. It's extremely frustrating. If anyone has found a solution, please point me in the right direction. Even advice on how to recycle memory more often. – scoofy Jul 26 '23 at 21:02

1 Answers1

2

There could be many reasons for this situation to be encountered. Is your app creating temporary files? Temporary files can be a cause of a memory leak. Temporary files can also be created from errors, or warnings. First of all, I would check my Stackdriver logs for errors and warnings and I would try to fix them.

Is your application interacting with databases or storage buckets ? Some memory related issues can be related to a bad interaction of your app with any data storage service. This issue was also encountered here and was mitigated by treating the Google Cloud Storage errors.

Another thing that you can do is investigate a bit the way of the memory is used in your function. For this you have some nice tools you can use like Pympler and Heapy. Playing with those tools may give you valuable clues about what your issue is.

Andrei Tigau
  • 2,010
  • 1
  • 6
  • 17