Desktop screen to Plane in Unity3D (RAM overflow)

Question

I try to stream the desktop of a Computer ingame onto a plane in Unity3D (and later interacting with it). I got to work the screencapturing, but already after a minute the unity game just fills the RAM of my PC (~30GB).

I already try to clean it by calling the Garbage Collector manually, but calling it doesn't seem to change anything.

I also wrote a test application in native C# in Visual Studio with Windows Forms. In the C# test application the code behaves all good and doesn't flood the RAM.

Is there a possibility, that Unity doesn't know how to free RAM? To get this code working I had to add the System.Drawing.dll to my Unity game.

If this approach is not working, are there any other options to stream the desktop and show it on a plane in Unity?

Here is my current code:

using System.Collections;
using System.Collections.Generic;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;
using System.Threading;
using UnityEngine;

public class ScreenCap : MonoBehaviour {

    public Renderer renderer;
    public MemoryStream stream;
    // Use this for initialization
    void Start () {
        renderer = GetComponent<Renderer>();
        startScreenCaptureThread();
    }

    // Update is called once per frame
    void Update () {

        Texture2D tex = new Texture2D(200, 300, TextureFormat.RGB24, false);

        if (stream != null)
        {
            tex.LoadImage(stream.ToArray());

            renderer.material.mainTexture = tex;
            stream.Dispose();
            System.GC.Collect();
            System.GC.WaitForPendingFinalizers();
        }
    }

    public void startScreenCaptureThread()
    {
        Thread t = new Thread(ScreenCapture);
        t.Start();
    }

    public void ScreenCapture()
    {
        Rectangle screenSize;
        Bitmap target;
        MemoryStream tempStream;
        while (true)
        {

            System.Diagnostics.Process proc = System.Diagnostics.Process.GetCurrentProcess();
            Debug.Log(proc.PrivateMemorySize64);
            screenSize = System.Windows.Forms.Screen.PrimaryScreen.Bounds;
            target = new Bitmap(screenSize.Width, screenSize.Height);
            using (System.Drawing.Graphics g = System.Drawing.Graphics.FromImage(target))
            {
                g.CopyFromScreen(0, 0, 0, 0, new Size(screenSize.Width, screenSize.Height));
            }
            tempStream = new MemoryStream();
            target.Save(tempStream, ImageFormat.Png);
            tempStream.Seek(0, SeekOrigin.Begin);
            stream = tempStream;
            target.Dispose();
            tempStream.Dispose();
            System.GC.Collect();

            System.GC.WaitForPendingFinalizers();
        }
    }
}

You can try to move initialization of Texture2D, Rectangle, Bitmap and MemoryStream variables out of methods. I'm also not sure about Threading in Unity. I'm think it will be better to use coroutines. — Woltus, May 26 '17 at 11:01
I am trying to add a virtual desktop in my VR project and found that using CopyFromSreen is subpar in terms of performance, and tops at 30fps, fully utilizing 1 CPU core. Looks like a better approach is to use DirectX libs like SharpDX, or smth like https://github.com/Phylliida/UnityWindowsCapture (could not yet figure out how it works) — Dmitrii Sutiagin, Dec 17 '17 at 19:11

score 2 · Accepted Answer · answered May 26 '17 at 12:12

When new Texture2D is created, it is never destroyed until you load another scene. Unity will then clean up the resources used by the Texture2D.

From your code, you are creating new Texture2D every frame. That indeed should be the biggest issue in your code.

Just move

Texture2D tex = new Texture2D(200, 300, TextureFormat.RGB24, false);

into the Start function then make the tex variable a public variable so that it can be used in the Update function. This way, you are not creating new Texture2D every frame.

Texture2D tex;

void Start () 
{
    tex = new Texture2D(200, 300, TextureFormat.RGB24, false);
}

void Update () 
{
    if (stream != null)
    {
        tex.LoadImage(stream.ToArray());
        ....
        ...
    }
}

Note:

There are big unrelated problems in your code:

1.Your code is not Thread-safe in anyway. You should use Thread.MemoryBarrier or the lock keyword. This is a long topic to discuss here. You can find many resources on the internet.

2.You are blindly updating the Texture2D without knowing if there is a new screenshot or not. You can fix that by calling the loading codes that is inside the Update function from the ScreenCapture() function.

Of-course, you will get

xxx can only be called from the main thread

exception. Use the UnityThread.executeInUpdate function this UnityThread class.

Place the code below to the end of the while loop:

UnityThread.executeInUpdate(() =>
{
    if (stream != null)
    {
        tex.LoadImage(stream.ToArray());

        renderer.material.mainTexture = tex;
        stream.Dispose();
        System.GC.Collect();
        System.GC.WaitForPendingFinalizers();
    }
});

You still have to do #1 to make it Thread-safe. I will leave that to you. You only need to make the stream variable Thread-safe since it's being used from different threads.

Thanks for the help, this helped me a lot. – xStream May 26 '17 at 13:04 — xStream, May 26 '17 at 13:04

Desktop screen to Plane in Unity3D (RAM overflow)

1 Answers1