-1

I am trying to understand how the following would work from a technical point of view...

Im constantly receiving bytes (essentially streaming video) for a video that is H.264 encoded. The problem im failing to see the solution for is that the bytes are retrieved programmatically (via UDP packets). In all the libraries i've seen which helps to load a stream and render it somehow, they all require a Uri as a source.

So wondering how to render an H.264 video that is constantly being received in raw bytes from a source that has no Uri?

This is why in a prior post (which has been voted to be deleted) I mentioned how i am trying to convert a single H.264 encoded video frame into a bitmap image, and subsequently rendering that bitmap image to a WriteableBitmap object (WPF).

Update

I will be trying out this when i get a chance to test it out.

AlvinfromDiaspar
  • 6,611
  • 13
  • 75
  • 140
  • 1
    Decoding and displaying video is independent of how it’s delivered. USD, TCP, HTTP, local file, pipe. It does t matter. – szatmary Jan 28 '20 at 01:53
  • @szatmary - are you saying that i can use "udp://" as my Uri? – AlvinfromDiaspar Jan 28 '20 at 02:07
  • 1
    Your url in what? You don’t mention any libraries or programs. I am assuming you are writing some software. I am saying the decode and playback component should be agnostic to the media source component. – szatmary Jan 28 '20 at 02:16
  • 1
    If you are trying into use a library or a program, then there is no possible way for anyone to tell you what protocol it supports without you telling use what is is. – szatmary Jan 28 '20 at 02:17
  • 1
    Take a look at ffmpeg it has support for dozzens of protocols and media types. – szatmary Jan 28 '20 at 02:17
  • i didnt mention any library because i wanted to simply know if it is even feasible to stream a video from a udp://ipaddress:port location. – AlvinfromDiaspar Jan 28 '20 at 02:48
  • 1
    First [stream in your data and convert it](https://stackoverflow.com/questions/7665217/how-to-process-raw-udp-packets-so-that-they-can-be-decoded-by-a-decoder-filter-i/7668578) to an RGB buffered image (I'd probably do this in unmanaged C++ myself for performance). Then subscribe to [CompositionTarget.Rendering](https://learn.microsoft.com/en-us/dotnet/framework/wpf/graphics-multimedia/how-to-render-on-a-per-frame-interval-using-compositiontarget) to invalidate a UserControl and render the buffer in its OnRender function. – Mark Feldman Jan 28 '20 at 04:15
  • @MarkFeldman - the stream is constantly sending me frames of the video. are you saying do the conversion and rendering on a per frame basis? This is what i originally wanted to do, but i got stuck when i couldnt convert a H.264 video frame to a bitmap image. – AlvinfromDiaspar Jan 28 '20 at 04:17
  • 1
    Yep. Have a look at [my answer to this question](https://stackoverflow.com/a/53999631/1177147) for an example of per-frame rendering in WPF. – Mark Feldman Jan 28 '20 at 04:23

1 Answers1

1

There is a long way between UDP packets and a displayable frame extracted from those packets. The steps are:

  1. UDP reception
  2. packet ordering, removal of duplicates, identification of lost packets
  3. depacketizing into H.264 bitstream (H.264 ES)
  4. use of H.264 video decoder
  5. fitting of decoder output to match your needs (texture vs. bytes, conversion to RGB and alike)

The steps are all non-trivial in their own way and in most cases you don't deal with them directly. In most cases there is some assumption and well-known protocol used, then you use existing piece of software to do the job for you taking into account that there is an agreement, such as video streaming per well known specification, and this software is capable to process data respectively.

For example, UDP packets might be structured according to real-time protocol (RTP). The protocol itself is defined by respective A Transport Protocol for Real-Time Applications document, then there is another document RTP Payload Format for H.264 Video for H.264 specifically, then it is assumed that the data is submitted to platform services for H.264 decoding, such as H.264 Video Decoder and so finally there is something approaching to be a video frame in the format you want it.

In all the libraries i've seen which helps to load a stream and render it somehow, they all require a Uri as a source.

Since the libraries (controls, sometimes high level APIs) would typically sole some problem end to end, they are covering streaming needs for all mentioned steps 1-5. You provide a network location (Uri) where the data comes from and can be received, then the library displays the data in user friendly way, displayed video that is.

Your initial description of the problem you are trying to solve

I am constantly receiving bytes (essentially streaming video) for a video that is H.264 encoded...

has no mention of the format of the data. This important and essential part alone makes it impossible to get you any solution for the problem. If you know the details for format and protocol, you mention it and you look for solution exactly for it. If you don't you have to find it and either deal with it yourself such as solving steps 1-3 in your own way and fit the data to more or less standard form that steps 4-5 could be solved in a regular way. If you know the format and there are platform and/or third party solutions for this type of data stream, then you again mention the type of data and you would probably want to use such solutions in your application.

Community
  • 1
  • 1
Roman R.
  • 68,205
  • 6
  • 94
  • 158
  • thanks for a very good answer. I thought it made it clear in my question: the protocol is UDP. The format is H.264. – AlvinfromDiaspar Jan 28 '20 at 17:05
  • See the steps I outlined above. I gave an example for UDP and H.264, and there are still more of degrees of freedom, such as items 2 and 3. Just H.264 over UDP is not describing sufficiently what is going on. If it's raw H.264 over UDP (which is not normally happens in real life) then you are responsible to handle steps 1-3 yourselves and build a valid H.264 ES stream at the very least. – Roman R. Jan 28 '20 at 18:10