2

I am currently working on a Unity project that runs on Windows and Android. I am basically using the Unity WebCamTexture class to access a connected camera and then send the image to my C++ code (dll on Windows, so on Android) which works with opencv for further processing.

While everything works fine on Windows, the exact same code is causing memory leaks on Android (RAM usage is constantly increasing on each frame). I have looked for several possible solutions on how to share the image with my C++ code. Each solution works fine on Windows and causes memory leaks on Android (the image processing works on Android too, just the leak is an issue)

Here is one working code example:

C#

    Color32[] color = webcamTexture.GetPixels32();
    byte[] data = Color32ArrayToByteArray(color);
    IntPtr unmanagedArray = Marshal.AllocHGlobal(data.Length);
    Marshal.Copy(data, 0, unmanagedArray, data.Length);

    PrepareImage(unmanagedArray, camW, camH);

    Marshal.FreeHGlobal(unmanagedArray);

C++

    void PrepareImage(unsigned char* data,int w, int h){

    cv::Mat textureC4(h, w, CV_8UC4, data);

    cv::Mat textureC3;

    cv::cvtColor(textureC4,textureC3,cv::COLOR_RGBA2RGB);

    }

The memory leak starts as soon as I am trying to make any changes to the "texture" variable in the C++ part (such as changing from 4 channels to 3).

Since I am not experienced in C++, I am not surprised to run in such issues, but what confuses me the most is that it runs perfectly fine on windows. I'll be happy for any suggestions pointing me into a new direction. Thank you!

Update:

A few more details to the environments:

Unity Version: 2018.2.8f1

Tested NDK versions: 13b and 16b

Tested Phones: 1x Samsung S9, 2x Note 8

These additional solutions have also been tested and give the same results:

Link 1

Link 2

Furthermore I have also tried to create the pointer on the C++, let C# use that pointer and let finally C++ clean up the memory. This is causing still the same leaks. At this point I am wondering if it is a setup up problem so I am gonna create new projects and update about the results later

...I have now created a new Unity project that only runs the above code + the webcamtexture. Still the exact same behavior. At this point I suspect its the Android Studio project for my C++ code.

...I have now created a Visual Studio project to create the .so file. The memory is still leaking, even with an almost blank project.

NDom
  • 21
  • 4
  • Find out if the problem is the C# or C++ side of the code by simply commenting out the code in your C++ `PrepareImage` function and see if there is still mem leak. If there is then `Marshal.FreeHGlobal` is likely failing to free the mem – Programmer Oct 16 '18 at 06:51
  • Thank you for your answer. The memory leak happens on the C++ side, when calling "cvtColor" or other functions that are trying to use the "textureC4" variable. " cv::Mat textureC4(h, w, CV_8UC4, data)" does not create any leaks. – NDom Oct 16 '18 at 07:00
  • Are you sure it is memory leak? Other possible causes that I have seen is: 1. the program just dirtifies some memory frames, but they are not actually allocated. The res memory appears to grow but it is OK, as it is released as soon as some other app needs it. 2. fragmentation, may be a bigger problem, you may have to replace your malloc with some custom library. – Yuri Nudelman Oct 16 '18 at 07:37
  • If there is a memory leak the memory usage will constantly grow at same speed until the app crashes. If not it will grow until some point, then slow down or even stop. So what is happening in your case? – Yuri Nudelman Oct 16 '18 at 07:39
  • Thank you for your answer. Unfortunately, it grows till it crashes. Per frame it adds a few MB to the RAM which is probably the image getting saved over and over again. – NDom Oct 16 '18 at 07:47
  • I know you said the problem is not `Marshal.AllocHGlobal` but what if you don't use `Marshal.AllocHGlobal` at-all? Pin the texture array then send to C++. See [this](https://stackoverflow.com/a/52687264/3785314) for example. Update me on the issue – Programmer Oct 16 '18 at 12:28
  • What do profilers say (e.g. Deleaker)?.. Also I would try to make a clean C++ sample to check if the problem is really on C++ side. – Artem Razin Oct 16 '18 at 16:15
  • @Programmer Unfortunately I have already tried that solution with the same results. I have also tried this idea [link](https://stackoverflow.com/questions/47381893/passing-images-to-opencv-from-unity), which uses a struct. I have even tried to create the pointer on the C++ side, send it over to C# fill the data there and then clean it up on C++. Everything has the same results: Memory leak after accessing the data on C++. – NDom Oct 17 '18 at 02:54
  • And you're saying that the issue is not there when you remove `cv::cvtColor`? – Programmer Oct 17 '18 at 03:24
  • @Programmer exactly. Though it is not specifically cv::cvtColor that makes the leak. I tried to change the channels from 4 to 3 without opencv, as I first thought that there might be a problem with the function. But even without opencv, as soon as I am trying to copy or change the data, I am getting memory leaks. cv::Mat textureC4(h, w, CV_8UC4, data) seems to be okay since I guess it is only pointing to the data. – NDom Oct 17 '18 at 04:07
  • Ok. I want you to try just the LINK2 you added to your question(which is my answer). Now, tell me which line is causing the allocation. – Programmer Oct 17 '18 at 08:33
  • @Programmer In your answer, the line `unsigned char* pixel = raw + (y * width * 4 + x * 4);` is causing the memory leak. Furthermore, after that I have also created the .so file in visual studio... just to make sure it is not Android Studio. ...and it is still leaking. – NDom Oct 18 '18 at 08:00
  • What's your Unity version? What's your Scripting run time version and Scripting Backend set to from the build settings? Finally, instead of passing camera data array to the C++ side, can you manually create your own array and pass it to the C++ side then see if there's still a leak? Remove or comment out the camera code for this test. Let me know the outcome. – Programmer Oct 18 '18 at 10:59
  • @Programmer I have tested the Unity Versions 2018.2.8f1 and 2017.4.13f1 LTS. I have created a color32 array with 320*240 length and filled it manually. Unfortunately still leaking(camera code was deleted). The scripting run time is .NET 3.5 Equivalent (I tested the change to .NET 4.x but it is still leaking). Scripting Backend is Mono. Api Compatibility Level is .NET 2.0 Subset – NDom Oct 19 '18 at 00:49

1 Answers1

0

In the end, it was a problem with the Android Studio project. I created a new project on a 3rd computer. I copied the exact same code. Now it all runs fine without leaks. Thank you to everyone who helped me! :)

NDom
  • 21
  • 4