0

I want to live stream my screen to a web page, I have a win forms that sends a screenshot of my screen, then I have a asp.net website that receives the img and puts it in a asp:Image tag. at first I wrote this code on two winforms application and it worked, but when I tried to transfer it to asp.net it doesn't work anymore.

//this is the win forms app
BinaryFormatter binFormatter = new BinaryFormatter();
    MemoryStream ms = new MemoryStream();
    while (connected)
    {
        stream = client.GetStream();
        System.Drawing.Image bmp = (System.Drawing.Bitmap)binFormatter.Deserialize(stream);
        bmp.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
        img.ImageUrl = "data:image/gif;base64," + Convert.ToBase64String(ms.ToArray());
        ms.FlushAsync();
    }
}







//this is the web page
private void ReciveImg()
{
    BinaryFormatter binFormatter = new BinaryFormatter();
    MemoryStream ms = new MemoryStream();
    while (connected)
    {
        stream = client.GetStream();
        System.Drawing.Image bmp = (System.Drawing.Bitmap)binFormatter.Deserialize(stream);
        bmp.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
        img.ImageUrl = "data:image/gif;base64," + Convert.ToBase64String(ms.ToArray());
        ms.FlushAsync();
    }
}
  • This seems like calling ms.ToArray() so often would dump a bunch of stuff into the large object heap that won't get cleaned up properly, and eventually cause an OOM situation due to lack of address space cleanup. – Joel Coehoorn May 26 '21 at 18:58
  • Also, can you clarify what you mean by "Doesn't work anymore"? – Joel Coehoorn May 26 '21 at 19:00
  • Joel Coehoorn, sorry didn't mean didn't work anymore. what I meant that when I translated it from two separate winforms application (one sharing screen and one displaying the screen) this code worked (but instead of displaying the img in , I displayed the imgs in a picturebox, I also didn't use the memory stream because there was no use to converting the bmp to base 64 string) hope this makes more sense, english isn't my first language thanks – גולני פיתוח May 26 '21 at 19:17
  • Also, Joel Coehoorn, is there any substitue for calling ms.ToArray often? and shouldn't ms.FlushAsync() clear the data? – גולני פיתוח May 26 '21 at 19:56
  • If you read the documentation for MemoryStream.FlushAsync(), you'll find it says the method is redundant: that is, it's included for compatibility with other APIs, but it doesn't do anything the garbage collector isn't already doing. In specific, the garbage collector **will** clean up the memory, but if these arrays are 85,000 bytes or larger (very common for jpeg and bmp objects) they will go on the large object heap and the garbage collector will **NOT** reclaim _Process Address Space_ the memory previously occupied, or which you are usually limited to 2GB regardless how much RAM you have – Joel Coehoorn May 26 '21 at 20:03
  • (continued) ... the normal fix here is to make sure the memory stream is writing into the **same array** each time. The array must be provisioned to be large enough for the biggest image file you will need, and you must track how many bytes you are using for the current image in the array manually. – Joel Coehoorn May 26 '21 at 20:06
  • In regard to the memory stream, the `flushAsync` si being used to clear the buffer and reset the position to 0. A more efficient way to do this might be to call `mas.Position = 0; ms..SetLength(0);` as explained here. https://stackoverflow.com/questions/2462391/reset-or-clear-net-memorystream It is also my understanding that the 85k issue is only prior to .net 4.5 https://stackoverflow.com/questions/12002443/how-do-i-force-release-memory-occupied-by-memorystream – Alexander Higgins May 26 '21 at 20:25

2 Answers2

0

ASP:Image implies a web forms project. I need to make sure you understand the basics of web forms as they differ from winforms: mainly, in web forms the form is destroyed as soon as the initial render is finished. Any properties or attributes of the form — like a timer to update an image control — doesn't have a long-term lifetime in a web forms environment; it's created, the form is rendered to the browser, and then it's immediately destroyed.

Every time you have a server event like a button click, the form is rebuilt from scratch, rendered to the browser, and then immediately destroyed again. If that sounds slow and expensive to you, you're right, and good web forms developers spend a lot of effort avoiding raising server events.

In other words, web forms does not save you from needing to know what's going on with the html and javascript used by the web browser. In this case, you want to write javascript code to refresh the image element in the html page.

Joel Coehoorn
  • 399,467
  • 113
  • 570
  • 794
  • thanks! but I still do not understand why it doesn't display anything, even tho it's slow and expensive isn't it still supposed to render the img? Because you said every time theres a server event the form is rebuilt from scratch, doesn't that mean that it'll at least show the imgs that I'm trying to display? – גולני פיתוח May 26 '21 at 19:54
  • In a web page, an image tag has it's own `src` attribute that is loaded as a separate http request to the server. By the time this http request is made, the form is already gone. – Joel Coehoorn May 26 '21 at 20:00
  • Also, server events are only raised from the user/browser. Think button clicks. A timer tick or elapsed event won't fire, because the timer responsible for it was destroyed. – Joel Coehoorn May 26 '21 at 20:09
0

Windows forms and Web forms have two different architectures.

In a windows form, when you modify an image on a control this happens in memory and the framework automatically takes care of repainting the image in the form.

In a web form, you have a client the performs an http request which the server receives and executes code according to the parameters of the request. Once all of the code is execute and http response is returned containing the data output as a result of your code.

In your case, the browser will send the http request and the server will execute the ReciveImg method which depending on how connected is implemented might run forever without ever even returning a response to the browser. Even if it does return a response, it would only return the last image you wrote.

To accomplish what you are attempting you need a different approach.

  1. You could have a method which returns a single image for each call. Then within the browser you could repeatedly call that method, perhaps using the legacy AjaxControlToolkit Asp:UpdatePanel configured to refresh as soon as an image is loaded or alternately using a JavaScript loop to repeatedly call your method. This approach can be complicated and not very efficient.

  2. A better option would be to set up a websocket between your client and your server. Once the client connects to the websocket the server can then begin writing images which the client will receive on each message received event. Then within javascript you can set the image to image received from the server. This approach is simpler and more efficient.

For an example of how to implement this, see my repo here which takes screen shots of a desktop and writes them to a websocket on the server side.

Server code:

class Program
    {
        static bool running = false;
        static void Main(string[] args)
        {
            Console.WriteLine("Press any key to start the WebSocketServer!");

            Console.ReadKey();
            Console.WriteLine();

            var appServer = new WebSocketServer();

            //Setup the appServer
            if (!appServer.Setup(2012)) //Setup with listening port
            {
                Console.WriteLine("Failed to setup!");
                Console.ReadKey();
                return;
            }

            appServer.NewMessageReceived += new SessionHandler<WebSocketSession, string>(appServer_NewMessageReceived);
            appServer.NewSessionConnected += AppServer_NewSessionConnected;
            Console.WriteLine();

            //Try to start the appServer
            if (!appServer.Start())
            {
                Console.WriteLine("Failed to start!");
                Console.ReadKey();
                return;
            }
            running = true;
            Console.WriteLine("The server started successfully, press key 'q' to stop it!");

            while (Console.ReadKey().KeyChar != 'q')
            {
                Console.WriteLine();
                continue;
            }

            //Stop the appServer

            running = false;
            Console.WriteLine();
            Console.WriteLine("The server was stopped!");
            Console.ReadKey();
        }

        private static void AppServer_NewSessionConnected(WebSocketSession session)
        {

            byte[] bytes = null;
            var fps = 1000 / 12.0;
            while (running && session.Connected)
            {
                DateTime start = DateTime.Now;
                bytes = ScreenShotUtility.TakeScreenshot();
                var segment = new ArraySegment<byte>(bytes);
                session.Send(segment);
                var diff = DateTime.Now.Subtract(start).TotalMilliseconds;
                var sleep = Math.Max(0, fps - diff);
                System.Threading.Thread.Sleep((int)sleep);
            }

        }

        static void appServer_NewMessageReceived(WebSocketSession session, string message)
        {
            //Send the received message back
            session.Send("Server: " + message);
        }
    }

Client code (using query and a websocket):

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
    <title>Test</title>
    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript">
        var noSupportMessage = "Your browser cannot support WebSocket!";
        var ws;
        var canvas;
        var ctx;
        var urlCreator;
        function wsMessage(e) {
            if (typeof e.data === "string") {
                appendMessage("# " + evt.data + "<br />");

            }
            else if (e.data instanceof ArrayBuffer) {

            } else if (e.data instanceof Blob) {
                //showBlob(e.data);
                blob2canvas(e.data);
            }
        }
        function appendMessage(e) {
            $('#lastMessage').html(e);
        }

        function blob2canvas(blob) {
            var img = new Image();
            img.onload = function () {
                if (canvas.width != img.width)
                    canvas.width = img.width;
                if (canvas.height != img.height)
                    canvas.height = img.height;
                ctx.drawImage(img, 0, 0);
            }
            img.src = urlCreator.createObjectURL(blob);// blob;
        }
        function showBlob(blob) {
            //console.log(blob);

            var imageUrl = urlCreator.createObjectURL(blob);
            document.querySelector("#blobImage").src = imageUrl;
            //disconnectWebSocket();
        }
        function connectSocketServer() {
            var support = "MozWebSocket" in window ? 'MozWebSocket' : ("WebSocket" in window ? 'WebSocket' : null);

            if (support == null) {
                appendMessage("* " + noSupportMessage + "<br/>");
                return;
            }

            appendMessage("* Connecting to server ..<br/>");
            // create a new websocket and connect
            var host = "localhost"; //"192.168.1.170"; //"localhost" <-- broke for edge, so is "127.0.0.1";
            ws = new window[support]('ws://'+host +':2012/');

            // when data is comming from the server, this metod is called
            ws.onmessage = function (evt) {
                wsMessage(evt);
            }
            ws.onerror = function (event) {
                console.error("WebSocket error observed:", event);
            };

            // when the connection is established, this method is called
            ws.onopen = function () {
                appendMessage('* Connection open<br/>');
                $('#messageInput').attr("disabled", "");
                $('#sendButton').attr("disabled", "");
                $('#connectButton').attr("disabled", "disabled");
                $('#disconnectButton').attr("disabled", "");
            };

            // when the connection is closed, this method is called
            ws.onclose = function () {
                appendMessage('* Connection closed<br/>');
                $('#messageInput').attr("disabled", "disabled");
                $('#sendButton').attr("disabled", "disabled");
                $('#connectButton').attr("disabled", "");
                $('#disconnectButton').attr("disabled", "disabled");
            }
        }

        function sendMessage() {
            if (ws) {
                var messageBox = document.getElementById('messageInput');
                ws.send(messageBox.value);
                messageBox.value = "";
            }
        }

        function disconnectWebSocket() {
            if (ws) {
                ws.close();
            }
        }

        function connectWebSocket() {
            connectSocketServer();
        }

        window.onload = function () {
            $('#messageInput').attr("disabled", "disabled");
            $('#sendButton').attr("disabled", "disabled");
            $('#disconnectButton').attr("disabled", "disabled");
            canvas = document.getElementById('canvas');
            ctx = canvas.getContext('2d');
            urlCreator = window.URL || window.webkitURL;
        }

    </script>
</head>
<body>
    <input type="button" id="connectButton" value="Connect" onclick="connectWebSocket()" /> <input type="button" id="disconnectButton" value="Disconnect" onclick="disconnectWebSocket()" /> <input type="text" id="messageInput" /> <input type="button" id="sendButton" value="Send" onclick="sendMessage()" /> <br />
    <div id="lastMessage"></div>
    <img id="blobImage" />
    <canvas id="canvas" height="500" width="500"></canvas>
</body>
</html>
Alexander Higgins
  • 6,765
  • 1
  • 23
  • 41