Frame-by-frame MJPEG streaming with C#/ASP.NET MVC

Question

I have been trying to setup a MJPEG stream in ASP.NET. I want to retrieve a MJEPG stream from a URL, and send every frame that I get to every connected client. Examples that I have been able to find only read from a set file, instead of a continues stream from URL, and send the entire file through MultiStreamContent. Since I retrieve frame-by-frame, I cannot do this. I would like to know if it is possible to do what I want with ASP.NET MVC. I'm currently using AForge video to retrieve the MJPEG stream from a link. My code for the controller class:

using System.Net.Http;
using System.Web.Http;
using AForge.Video;

namespace VideoPrototypeMVC.Controllers
{
    public class CameraController : ApiController
    {
        int framecounter = 0;
        MJPEGStream stream = new MJPEGStream();

        [HttpGet]
        public void GetVideoContent()
        {   
            stream.Source = @"http://127.0.0.1:5002/stream";
            stream.NewFrame += new NewFrameEventHandler(showFrame);
            stream.Start();
            MultipartContent content = new MultipartContent();
            while (stream.IsRunning)
            {
            //Continues streaming should be here?
            }
        }

        //Can be used to display of a frame is available
        private void showFrame(object sender, NewFrameEventArgs eventArgs)
        {
            framecounter++;
            System.Diagnostics.Debug.WriteLine("New frame event: " + framecounter);

        }

        //Should be called at the end of the stream
        private void stopStream(object sender, ReasonToFinishPlaying reason)
        {
            System.Diagnostics.Debug.WriteLine("Stop stream");
            stream.Stop();
            framecounter = 0;
        }
    }
}

This code is not final, but I just need to get the continues streaming down. I have found examples that use Socket servers, but I would like to stick to MVC since it allows me to set up the rest of the server easier.

If I understand your question correctly, what you have is a device sending you a MJPEG stream and you want to provide the same MJPEG stream to multiple users connecting to your ASP.NET MVC application? You can do that if you share your MJPEG client accross all requests (via dependency injection, a static variable,...). Each request could just wait for a new frame, and send it to the client as it arrives. You can use MultiStreamContent to send the data to your clients; the connection will remain open (and idle) while you're waiting for more frames to arrive. — Frederik Carlier, Nov 13 '17 at 08:44
Check `PushStreamContent`. Then when first request arrives - start your MJPEG stream and write frames to all connected response streams. When last client disconnects - stop your MJPEG stream. — Evk, Nov 13 '17 at 08:55
Using dependency injection? I'll have to look into it, but it sounds like a proper possibility. Thanks. — Arastelion, Nov 13 '17 at 08:58
PushStreamContent, I have only found in the context of direct file streaming, does that also work when I pass a stream URL? @Evk — Arastelion, Nov 13 '17 at 09:03
You receive frame from `MJPEGStream` as byte array (I assume). Then you can just take this byte array and write it to the response of connected client. `PushStreamContent` allows you to do just that - write to the response stream in asynchronous manner (that is - write when your mvc method has already "returned"). You can write anything to response stream, not just files. — Evk, Nov 13 '17 at 09:06
@Evk That sounds exactly what I want. I will take a proper look at it. Thank you for the advice! — Arastelion, Nov 13 '17 at 09:10

score 3 · Accepted Answer · answered Nov 17 '17 at 14:54

To make sure other people will manage with this as well. I managed to combine what @Evk (Thank you once again) said, together with information I found here: creating my own MJPEG stream.

DO NOTE: The code below is just prototype/proof-of-concept! When I run this my processor shoots to 100% because of the endless while loop in StartStream. Will work on making this more event based but I thought the code below was more easely explained.

using System;
using System.IO;
using System.Net;
using System.Web;
using System.Net.Http;
using System.Web.Http;
using AForge.Video;
using System.Drawing;
using System.Text;
using System.Drawing.Imaging;
using System.Threading;

namespace VideoPrototypeMVC.Controllers
{
    public class CameraController : ApiController
    {
        private MJPEGStream mjpegStream = new MJPEGStream();
        private bool frameAvailable = false;
        private Bitmap frame = null;
        private string BOUNDARY = "frame";

        /// <summary>
        /// Initializer for the MJPEGstream
        /// </summary>
        CameraController()
        {
            mjpegStream.Source = @"{{INSERT STREAM URL}}";
            mjpegStream.NewFrame += new NewFrameEventHandler(showFrameEvent);
        }

        [HttpGet]
        public HttpResponseMessage GetVideoContent()
        {   
            mjpegStream.Start();
            var response = Request.CreateResponse();
            response.Content = new PushStreamContent((Action<Stream, HttpContent, TransportContext>)StartStream);
            response.Content.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse("multipart/x-mixed-replace; boundary=" + BOUNDARY);
            return response;
        }

        /// <summary>
        /// Craete an appropriate header.
        /// </summary>
        /// <param name="length"></param>
        /// <returns></returns>
        private byte[] CreateHeader(int length)
        {
            string header =
                "--" + BOUNDARY + "\r\n" +
                "Content-Type:image/jpeg\r\n" +
                "Content-Length:" + length + "\r\n\r\n";

            return Encoding.ASCII.GetBytes(header);
        }

        public byte[] CreateFooter()
        {
            return Encoding.ASCII.GetBytes("\r\n");
        }

        /// <summary>
        /// Write the given frame to the stream
        /// </summary>
        /// <param name="stream">Stream</param>
        /// <param name="frame">Bitmap format frame</param>
        private void WriteFrame(Stream stream, Bitmap frame)
        {
            // prepare image data
            byte[] imageData = null;

            // this is to make sure memory stream is disposed after using
            using (MemoryStream ms = new MemoryStream())
            {
                frame.Save(ms, ImageFormat.Jpeg);
                imageData = ms.ToArray();
            }

            // prepare header
            byte[] header = CreateHeader(imageData.Length);
            // prepare footer
            byte[] footer = CreateFooter();

            // Start writing data
            stream.Write(header, 0, header.Length);
            stream.Write(imageData, 0, imageData.Length);
            stream.Write(footer, 0, footer.Length);
        }

        /// <summary>
        /// While the MJPEGStream is running and clients are connected,
        /// continue sending frames.
        /// </summary>
        /// <param name="stream">Stream to write to.</param>
        /// <param name="httpContent">The content information</param>
        /// <param name="transportContext"></param>
        private void StartStream(Stream stream, HttpContent httpContent, TransportContext transportContext)
        {
            while (mjpegStream.IsRunning && HttpContext.Current.Response.IsClientConnected)
            {
                if (frameAvailable)
                {
                    try
                    {
                        WriteFrame(stream, frame);
                        frameAvailable = false;
                    } catch (Exception e)
                    {
                        System.Diagnostics.Debug.WriteLine(e);
                    }
                }else
                {
                    Thread.Sleep(30);
                }
            }
            stopStream();
        }

        /// <summary>
        /// This event is thrown when a new frame is detected by the MJPEGStream
        /// </summary>
        /// <param name="sender">Object that is sending the event</param>
        /// <param name="eventArgs">Data from the event, including the frame</param>
        private void showFrameEvent(object sender, NewFrameEventArgs eventArgs)
        {
            frame = new Bitmap(eventArgs.Frame);
            frameAvailable = true;
        }

        /// <summary>
        /// Stop the stream.
        /// </summary>
        private void stopStream()
        {
            System.Diagnostics.Debug.WriteLine("Stop stream");
            mjpegStream.Stop();
        }
    }
}

If you want to improve this and remove the endless loop: Use showFrameEvent. Any code in this is executed every time a frame is received. — Arastelion, Jan 04 '18 at 11:27
So the code above is that 'finished' seems to work very well - so thanks if so. — Richard Housham, Feb 27 '20 at 13:21

score 0 · Answer 2 · answered Mar 20 '20 at 09:42

Great answer from Arastelion however I noticed that if you leave the application then there is still a request going on in the background which can be a resource hog popping in stream.FlushAsync(); stream.Close(); stream.Dispose(); after the stopStream seems to resolve that.

using System;
using System.IO;
using System.Net;
using System.Web;
using System.Net.Http;
using System.Web.Http;
using AForge.Video;
using System.Drawing;
using System.Text;
using System.Drawing.Imaging;
using System.Threading;

namespace VideoPrototypeMVC.Controllers
{
    public class CameraController : ApiController
    {
        private MJPEGStream mjpegStream = new MJPEGStream();
        private bool frameAvailable = false;
        private Bitmap frame = null;
        private string BOUNDARY = "frame";

        /// <summary>
        /// Initializer for the MJPEGstream
        /// </summary>
        CameraController()
        {
            mjpegStream.Source = @"{{INSERT STREAM URL}}";
            mjpegStream.NewFrame += new NewFrameEventHandler(showFrameEvent);
        }

        [HttpGet]
        public HttpResponseMessage GetVideoContent()
        {   
            mjpegStream.Start();
            var response = Request.CreateResponse();
            response.Content = new PushStreamContent((Action<Stream, HttpContent, TransportContext>)StartStream);
            response.Content.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse("multipart/x-mixed-replace; boundary=" + BOUNDARY);
            return response;
        }

        /// <summary>
        /// Craete an appropriate header.
        /// </summary>
        /// <param name="length"></param>
        /// <returns></returns>
        private byte[] CreateHeader(int length)
        {
            string header =
                "--" + BOUNDARY + "\r\n" +
                "Content-Type:image/jpeg\r\n" +
                "Content-Length:" + length + "\r\n\r\n";

            return Encoding.ASCII.GetBytes(header);
        }

        public byte[] CreateFooter()
        {
            return Encoding.ASCII.GetBytes("\r\n");
        }

        /// <summary>
        /// Write the given frame to the stream
        /// </summary>
        /// <param name="stream">Stream</param>
        /// <param name="frame">Bitmap format frame</param>
        private void WriteFrame(Stream stream, Bitmap frame)
        {
            // prepare image data
            byte[] imageData = null;

            // this is to make sure memory stream is disposed after using
            using (MemoryStream ms = new MemoryStream())
            {
                frame.Save(ms, ImageFormat.Jpeg);
                imageData = ms.ToArray();
            }

            // prepare header
            byte[] header = CreateHeader(imageData.Length);
            // prepare footer
            byte[] footer = CreateFooter();

            // Start writing data
            stream.Write(header, 0, header.Length);
            stream.Write(imageData, 0, imageData.Length);
            stream.Write(footer, 0, footer.Length);
        }

        /// <summary>
        /// While the MJPEGStream is running and clients are connected,
        /// continue sending frames.
        /// </summary>
        /// <param name="stream">Stream to write to.</param>
        /// <param name="httpContent">The content information</param>
        /// <param name="transportContext"></param>
        private void StartStream(Stream stream, HttpContent httpContent, TransportContext transportContext)
        {
            while (mjpegStream.IsRunning && HttpContext.Current.Response.IsClientConnected)
            {
                if (frameAvailable)
                {
                    try
                    {
                        WriteFrame(stream, frame);
                        frameAvailable = false;
                    } catch (Exception e)
                    {
                        System.Diagnostics.Debug.WriteLine(e);
                    }
                }else
                {
                    Thread.Sleep(30);
                }
            }
            stopStream();
            stream.FlushAsync();
            stream.Close();
            stream.Dispose();
        }

        /// <summary>
        /// This event is thrown when a new frame is detected by the MJPEGStream
        /// </summary>
        /// <param name="sender">Object that is sending the event</param>
        /// <param name="eventArgs">Data from the event, including the frame</param>
        private void showFrameEvent(object sender, NewFrameEventArgs eventArgs)
        {
            frame = new Bitmap(eventArgs.Frame);
            frameAvailable = true;
        }

        /// <summary>
        /// Stop the stream.
        /// </summary>
        private void stopStream()
        {
            System.Diagnostics.Debug.WriteLine("Stop stream");
            mjpegStream.Stop();
        }

         protected override void Dispose(bool disposing)
         {
            if (disposing)
             {
                 stopStream();
             }

             base.Dispose(disposing);
           } 


    }
}

Frame-by-frame MJPEG streaming with C#/ASP.NET MVC

2 Answers2

Linked