How to play H264 frames on a web page?

Question

My security camera provides frames in H264 codec via RTSP. Below is my C++ code with FFMPEG library getting frames from from the camera.

std::string read_frame() {
    AVPacket *packet;
    packet= av_packet_alloc();

    if (!packet) return "";
    
    while (av_read_frame(format_context, packet) >= 0) {
        if (packet->stream_index != stream_index) continue;
    
        if (avcodec_send_packet(codec_context, packet) < 0) return "";
   
        fflush(stdout);

        int response = avcodec_receive_frame(codec_context, frame);

        if (response == AVERROR(EAGAIN) || response == AVERROR_EOF) {
            continue;
        } else if (response < 0) {
           return "";
        }

        // std::cout << codec_context->codec_id << " == " << AV_CODEC_ID_H264 << std::endl;

        // std::string str = decode_base64(encode_base64(packet->data, packet->size)); // I did try this to get binary data from AV Packet
        std::string str = decode_base64(encode_base64(frame->data, (frame->width * frame->height))); // I did try this to get binary data from AV Frame.

        fflush(stdout);

        return str;
    }
}

I use Base64 encode function to extract the frame data out of the AV Packet or AV Frame and then use the Base64 decode function to get raw binary data as a string. My web socket pushes this raw binary data (H264) to the web page.

Due to this answer: Play raw h264 live stream in browser. Below is my JavaScript code that connects to the web socket.

script type="text/javascript" src="dist/jmuxer.min.js"></script>

<video id="player"></video>

<script>
    var jmuxer = new JMuxer({
        node: 'player',
        mode: 'video',
        debug: true
    });

    const ws = new WebSocket('ws://127.0.0.1:port_number', ["binary"]);

    ws.addEventListener('message', event => {
        event.data.arrayBuffer().then(res => {
            jmuxer.feed({
                video: new Uint8Array(res)
            });
        });
       
        // Have also tried this, but did not work.
        // jmuxer.feed({
            // video: new Uint8Array(event.data)
            //// video: new Uint8Array(event.data.arrayBuffer()) // Also tried this.
        //});
    });
</script>

My web socket does work well since I do receive binary data on the web page. No doubt about that. My read_frame() C++ function also works since I do see binary data with std::cout << read_frame() << std::endl; and it is in H264 codec std::cout << codec_context->codec_id << " == " << AV_CODEC_ID_H264 << std::endl;

How I can show frames on the web page with the H264 binary data? Any other JavaScript libraries that you have ever worked with and you know it works well?

This sounds like a massive XY problem. Please [read through the answers here](https://stackoverflow.com/questions/1735933/streaming-via-rtsp-or-rtp-in-html5) especially [this one](https://stackoverflow.com/a/70400665/1548468) and [this one](https://stackoverflow.com/a/39697546/1548468) and see if any fit your situation. — Botje, Aug 04 '22 at 07:49
you have a problem Y "I want to see my security camera feed on a webpage" and decided that you need to solve problem X "I need to use FFMpeg and video-over-websockets" first. I pointed you to ready-made solutions for problem Y that do not involve problem X at all. See also the [XY problem website](https://xyproblem.info) — Botje, Aug 04 '22 at 08:07
@Botje There are some situations which you can't do the standard architecture that would make more sense. It's useful to signal to the questionner that he should think about the bigger picture. But sometimes, other constraints in the architecture can't prevent from using the "good" architecture. I have the same question and my architecture prevents me from doing rtsp. I need to manipulate a raw H264 to be imported into video element. — gregoiregentil, Apr 02 '23 at 22:23
Of course there might be situations (like yours?) that warrant doing nonstandard things. If this question had sufficient motivation and elaborated on the constraints I would not have the "this is an XY problem" reflex. I suggest you create a new question and see if anyone answers. — Botje, Apr 03 '23 at 07:25

How to play H264 frames on a web page?

0 Answers0