4

I have the following code which uses a third party library called Aspose.Words:

SaveOptions saveOptions = SaveOptions.CreateSaveOptions(SaveFormat.Docx);
saveOptions.MemoryOptimization = true;
saveOptions.TempFolder = Path.GetTempPath();
var mm = new RecyclableMemoryStreamManager();
using (RecyclableMemoryStream ms = new RecyclableMemoryStream(mm))
{
    doc.Save(ms, saveOptions);
    return ms.ToArray();
}

I hit the following error on the using statement.

Mvc.ExceptionHandling.AbpExceptionFilter - Exception of type 'System.OutOfMemoryException' was thrown.

I'm unable to reproduce it locally (my memory usage goes up by perhaps 200mb while its processing so it doesn't really use much memory). The file itself is only 56MB in size. I'm told by Aspose it could use up to 10 times that amount... even still it should be fine. I've altered the Service plan to scale up to 14GB of memory.

I'm using RecyclableMemoryStreamManager because I already tried it with MemoryStream to no avail.

What else could I do to problem solve this issue that I'm only hitting in Azure (dev & production)? Azure API is .net core stack and has a platform of 64bit.

Thanks in advance.

Sniipe
  • 1,116
  • 3
  • 13
  • 28

4 Answers4

6

What Azure Service offering are you using?

Most App Service based offerings, including Azure Functions in consumption plan maxxes out at 1.5GB memory per process/app regardless of the app service pricing plan you choose. Azure functions in consumption plan have a max memory limit of 1.5GB per function app instance (scaling can allow multiple instances) and other offerings also have limits to prevent you from gobbling up all the memory of the underlying machines. (Your local environment does not have these limits)

It is hard to solve without actual metrics. Run your code, then give it a few minutes after the crash and see metrics for the app, especially under Working Memory and private bytes. These are found in the metrics section in the App Service.

If these hit around the limits before your crash, this is likely the reason. If this is the case, your options in Azure are

  • Reduce your memory footprint
    • Optimize the current work so it uses less memory
    • Or split the work into multiple jobs which each uses less memory and use Azure Functions or similar
  • use an App Service Environment (lot of memory, isolated networking, but not cheap)
  • use a VM (Azure got memory optimized VMs for practically any size)
Madushan
  • 6,977
  • 31
  • 79
  • Thanks for the feedback - I will decouple to app an use Azure Functions. I hope there is a way to schedule deletions with something like Remove-AzureStorageFile (https://learn.microsoft.com/en-us/powershell/module/azure.storage/remove-azurestoragefile?view=azurermps-6.13.0) – Sniipe Dec 17 '19 at 10:46
  • 1
    btw I've updated the answer with limits. 1.5GB was apparently only for azure functions in consumption plan. But the other points are valid. – Madushan Dec 17 '19 at 11:07
2

I have recently dealt with this as we were storing a lot in cache using IMemoryCache. The Azure App Service hit 1.8 GB and it started to throw System.OutOfMemoryException even Azure Service Plan reached only 40% of memory used.

Here are steps to identify and correct this:

  1. Check which OS you are running on your App Service (App Service Plan should tell you), if Linux, there should be no problem.

enter image description here

  1. If you are running on Windows make sure it is set to 64 bit:

enter image description here

  1. Make sure your project is built with PlatoformTarget = AnyCpu

enter image description here

  1. Make sure you CI (like Azure Pipelines is not overwriting it)

  2. To confirm that your process is running in 64-bit, you can take memory dump in Diagnose and Solve Problems > Diagnostic Tools > Collect a Memory dump

enter image description here

Opening the memory dump report for your application tells you if your processing is running 32-bit or 64-bit. (Maybe there is another place this can be found easier but this one guarantees to confirm it)

enter image description here

Artur Kedzior
  • 3,994
  • 1
  • 36
  • 58
1

Scale up App Service Plan of your Azure service. Memory may be very less in App Service Plan. Also check if you have multiple services running in App Service Plan then scale out.

Do Scale up or Scale out.

Saurabh Raoot
  • 1,303
  • 3
  • 26
  • 31
  • I've tried scaling up - I'll try scale out now. thanks – Sniipe Dec 13 '19 at 16:15
  • If there is only one instance running (for my testing) do I need to scale out? – Sniipe Dec 13 '19 at 16:17
  • 1
    If multiple services are running then you need to scale out. Check the metrics "CPU Percentage" and "Memory Percentage" of your App service plan and decide – Saurabh Raoot Dec 13 '19 at 16:22
  • Oddly both the CPU and Memory percentage are very low and it doesn't spike. I am the only one using the environment and this is the only service running. To me the error shouldn't be outofmemory. – Sniipe Dec 13 '19 at 16:24
  • 4
    Also check if you are deploying 32 bit executable or 64 bit executable. If it is 32 bit it can not use the memory more than 2 GB, even if you have more RAM – Saurabh Raoot Dec 13 '19 at 16:25
  • 1
    https://stackoverflow.com/questions/1153702/system-outofmemoryexception-was-thrown-when-there-is-still-plenty-of-memory-fr?rq=1 – Saurabh Raoot Dec 13 '19 at 16:26
  • Thanks Saurabh, that's an interesting article. My azure api platform was set to 64bit but I might do some more digging around 32 vs 64. Perhaps the app is 32bit... – Sniipe Dec 13 '19 at 16:35
0

I did simply restarted my app service and then it is started working fine.

Muni Chittem
  • 988
  • 9
  • 17