0

Hello everyone I am seeing a major uptick in crashes regarding memory leaks in our recent Android builds. We have done some things to try to mitigate these issues, but still am seeing the same crashes in the latest release.

 Fatal Exception: java.lang.OutOfMemoryError
 Failed to allocate a 16 byte allocation with 1890136 free bytes and 1845KB until OOM, target footprint 201326592, growth limit 201326592; failed due to fragmentation (largest possible contiguous allocation 54788096 bytes)
 java.lang.Long.valueOf (Long.java:845)
 io.reactivex.internal.operators.observable.ObservableInterval$IntervalObserver.run (ObservableInterval.java:82)
 io.reactivex.Scheduler$PeriodicDirectTask.run (Scheduler.java:562)
 io.reactivex.Scheduler$Worker$PeriodicTask.run (Scheduler.java:509)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run (ExecutorScheduler.java:288)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run (ExecutorScheduler.java:253)
 java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
 java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
 java.lang.Thread.run (Thread.java:923)

 Fatal Exception: java.lang.OutOfMemoryError
 Failed to allocate a 16 byte allocation with 1590248 free bytes and 1552KB until OOM, target footprint 201326592, growth limit 201326592; failed due to fragmentation (largest possible contiguous allocation 39845888 bytes)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.schedule (ExecutorScheduler.java:161)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.schedule (ExecutorScheduler.java:187)
 io.reactivex.Scheduler$Worker$PeriodicTask.run (Scheduler.java:531)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run (ExecutorScheduler.java:288)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run (ExecutorScheduler.java:253)
 java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
 java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
 java.lang.Thread.run (Thread.java:923)

 Fatal Exception: java.lang.OutOfMemoryError
 Failed to allocate a 16 byte allocation with 1215008 free bytes and 1186KB until OOM, target footprint 201326592, growth limit 201326592; failed due to fragmentation (largest possible contiguous allocation 49020928 bytes)
 io.reactivex.internal.queue.MpscLinkedQueue.offer (MpscLinkedQueue.java:62)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.schedule (ExecutorScheduler.java:167)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.schedule (ExecutorScheduler.java:187)
 io.reactivex.Scheduler$Worker$PeriodicTask.run (Scheduler.java:531)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run (ExecutorScheduler.java:288)
 io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run (ExecutorScheduler.java:253)
 java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
 java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
 java.lang.Thread.run (Thread.java:923)

is there some framework change that is triggering these issues, is this application code that is causing this? what are some strategies to try to address crashes like the above?

Etienne Lawlor
  • 6,817
  • 18
  • 77
  • 89
  • 2
    OOM is caused by the entire app using too much memory. Forget about where you're seeing the crash- unless its a single huge allocation, it's not about the stack trace. It's about reducing memory usage overall. You actually have free memory left, but you hit fragmentation- you didn't have a big enough chunk of memory to make the allocation. That's really impressive, I've never seen that before. Basically you must be making tens of thousands or more tiny objects that it can't fit back together, but holding onto enough of them that it can't recombine them. – Gabe Sechan Jan 11 '23 at 07:05
  • 1
    I'd start with a stack dump and look for what objects beyond the basics (skip char[] and string because those are impossible to track down there's so many) are being created and held onto. I'd also use LeakCanary and see if it can track any leaks that may be holding those objects so it can't recombine them. – Gabe Sechan Jan 11 '23 at 07:07
  • I have used leak canary and the android studio memory profiler and have addressed some issues around memory leaks and references to variables that haven't been garbage collected. However I can't reproduce these errors on a debug build. These errors are happening on release builds and I'm not able to reproduce them but they are happening to our customers. – Etienne Lawlor Jan 12 '23 at 03:11
  • Any commonality to device or OS version it's occurring on? – Gabe Sechan Jan 12 '23 at 04:47
  • It's happening primarily on Android 11 Samsung tablets but that's only because that's our most popular configuration. It's also happened on other versions and other manufacturers. – Etienne Lawlor Jan 13 '23 at 03:41
  • If it is not due to OS and device issues, I will guess it is either due to loading oversize resources (e.g., raw bitmap in 4K without resize the resolution) or implementation issue in `ViewGroup` such as `RecyclerView`. Is there any dynamic resource that your app get from the internet? Probably just do a `git diff` to compare it with a stable build. – Android Newbie A Jan 18 '23 at 05:04
  • I think the difference between you and your users is that they are probably working with the app much longer then you do and they do different usecases in your app. If you have some memory leaks, they add up over time, until the memory is sucked out - if you only try around for 20 minutes, it might not be long enough to fill up your whole memory. You maybe need some app logs + leak canary, to see what your users do over multiple hours to try to recreate the situation. The tricky part for the memory leaks is that it takes sometimes hours because they sneak in step by step – Radu M Jan 18 '23 at 10:28

3 Answers3

0

Some other techniques to consider beyond the existing comments:

In field instrumentation:

Activity Patterns: If you have something that records user activity, look for apps that go a long time without crashing and apps that crash earlier and see if there are different actions performed by the user

Direct Memory Usage: Since you are not yet able to reproduce this on debug builds, you could record memory available just before and just after particular activities to help you narrow down where in the app this occurring. You can access app available memory and then log it (if you can get the logs) or report it back through some analytics system.

Local testing:

(with Leak Canary or profiler)

There are often points in time that should come back to the same level of allocated memory: for instance if you go into a screen and come back out you may allocate some static items, but from 2nd use of the screen onwards you will want the memory to come back to a normal (quiescent) point. So stopping execution, forcing a GC, restarting execution and going through a workflow and then coming back to the home screen. (again skipping the first time) Can be a good way to narrow down which workflow is leaving significant extra memory.

It is unusual that the debug builds are not producing this effect, if you have a "friendly" end user reporting this issue, perhaps give them a debug build and ask them to support you by using it.

In a debug environment you can also try to "make it worse" so, for example, go into and out of a screen or workflow 10 or 100 times (scripting for the 100 example).

Cadmium
  • 623
  • 3
  • 9
  • Additionally, when you app launches you can allocate a significant block of memory in your app and hang onto it - which can increase the chance you hit the out of memory condition (though as mentioned in a previous comment the stack at the time of OOM probably doesn't matter...) – Cadmium Jan 13 '23 at 15:02
0

Use Coroutines for long or heavy operations. These crashes are coming from Rxjava. Maybe you are not performing work accurately in that.

Sohaib Ahmed
  • 1,990
  • 1
  • 5
  • 23
0

I am just going to have a stab at where you could look, from the stack trace it looks like you are using a schedular to perform tasks. My suspicion is that you are running multiple threads, and as each thread requires its own allocation of memory some thing to consider would be: Controlling the number of threads through a thread pool, this will cap the number of threads available and recycle threads instead of allocating new ones and potentially having a significant number of threads running at the same time.

user2480415
  • 77
  • 2