How can I freeze a video MediaStreamTrack?

Question

I have a MediaStreamTrack that's streaming a video from the user's camera onto a <video> tag in Google Meets.

Running as a chrome extension, I'd like to replace it with a track that's just a static image.

I've tried to change the enabled flag as specified in other questions ( which turns it black instead of freezing the image ), calling stop() which kills it, or using ImageCapture.TakePhoto(...) to generate an image, put it in a canvas, and capture the canvas source - this failed because the canvas stream didn't match the original one, resulting in a black image. I tried to override other properties using Object.defineProperty() to replace the track with another after calling removeTrack(), didnt work either.

Question:
How would you go about "freezing" mediaStreamTrack so it contains, gets sent and recorded as just a static image?

(editor's note:
This part is from a comment on a (now deleted) Answer. It clarifies the Askers expected results)

Can you show me an example of how you'd get the source for the canvas from that (webcam) video before changing the stream for the output video into that canvas stream?

Here's the code that I used:

var vid = getVideoElementFromPage();
var Stream = vid.captureStream();
var Track = Stream.getVideoTracks()[0];
var URLIdea = undefined;
new ImageCapture(Track).takePhoto().then((imCpt) => {
    URLIdea = URL.createObjectURL(imCpt);
    // Here, I've tried setting this URL to a canvas, capturing stream and changing the track,
    // or changing the video src to the URL directly. 
    // in both cases - black screen
});

@Kaiido I'm not sure, could be. I'm trying to modify the google meets video. How can I test this? — A. Abramov, Apr 04 '23 at 14:45
How am I supposed to know what you need? But in the question you said the MediaStream is to be consumed by a — Kaiido, Apr 04 '23 at 23:59
@Kaiido I am trying to "freeze" the video element showing me in google meets for both myself (on the client) and the peers. I can take a photo with ImageCapture and put an image up locally to "freeze" the client - what I'm trying to do is modify the MediaStream so remote participants also see the frozen image. Let me know if there's anything else I can explain that would help, and thank you — A. Abramov, Apr 05 '23 at 11:05
So you don't even have control over the page's code. Are you running a user script or in the console, after the page (and the stream(s)) bave been initialized? — Kaiido, Apr 05 '23 at 11:20
@A.Abramov **(1)** Regarding your _"Not sure why the previous guy changed the tags"_, I changed them to meet the bounty requirement of _"**draw more attention** to this question"_. You had less than (typically expected for a bounty) views after 4 days. The [`video`](https://stackoverflow.com/questions/tagged/video) tag has 3.7 thousand watchers, your chosen [`mediastream`](https://stackoverflow.com/questions/tagged/mediastream) tag has only 10 watchers. It's more likely that someone who has experience in handling your problem is actually watching the `video` tag more than `mediastream` tag... — VC.One, Apr 05 '23 at 14:02
@A.Abramov **(2)** The hijacking of a Google Meets video stream is a new technical detail that you did not mention, See: first line of your Question text. That one is doable. Do you **still want** an Answer about freezing the frame of a mediastream in a ` — VC.One, Apr 05 '23 at 14:22
@VC.One Hi, I understand. I'm running within a chrome extension, will add this to the question — A. Abramov, Apr 07 '23 at 10:35
Instead of freezing the existing stream, have you considered creating a stream from scratch with a canva, proxy the webcam through this canva object where you can control everything and only use this canva as a source for your final stream to be sent ? This way, your "canva based stream" can display what you want, no need to freeze but just need to only display an image. But you will need to be able to set this canva as a webcam. — Jorgu, Apr 14 '23 at 08:01
@Jorgu I've tried using a canvas with a blob image as the source - as per the notes, the result is a black screen. — A. Abramov, Apr 15 '23 at 20:21
@KareemAdel Canvas stream results in a black screen in meets. — A. Abramov, Apr 15 '23 at 20:22
create a new `MediaStream` that contains only the new `MediaStreamTrack` and replace the original track with the new track in the original `MediaStream`, and replace the original MediaStream with the new stream — Kareem Adel, Apr 15 '23 at 20:28
@KareemAdel I created a new MediaStream that contains only the CanvasMediaStreamTrack, whereas the canvas source is the image blob I got from captureImage. the result was a black screen. — A. Abramov, Apr 15 '23 at 20:47
@A.Abramov Test this [online demo](https://vc-lut.000webhostapp.com/demos/public/camera_record_01/index.html). It temporarily only uses the Camera (but could load custom image or video etc). Do you think the output stream part can be plugged into Google Meet? — VC.One, Apr 16 '23 at 12:35

score 1 · Answer 1 · answered Apr 16 '23 at 12:17

Try testing this for now.
This code is "frankensteined"(?) from another (private code) project but I will update this demo code properly if you think it's closer to the solution that you want. Update means adding options to transmit a loaded image or video (via file picker).

PS: See code comments...

<!DOCTYPE html> 
<html> 
<body style="background-color: rgb(235, 235, 200)" /> 

<!-- select INPUT (use Camera) -->
(1) INPUT : <span id="txt_input"> using None </span> <br>
<button id="btn_input_image">Use Image</button>
<button id="btn_input_video">Use Video</button>
<button id="btn_input_webcam">Use Camera</button>

<br><br>

<div id="container_input" width="640">
<video id="myVideo" muted style="width:120px; height:90px;" >
</video>
</div>
<br>

<!-- the MIXER allows for INPUT to be edited before sending to OUT -->
(2) MIXER : <br>
<button id="btn_fx_pause">Pause video</button>
<button id="btn_fx_color1">Effect video</button>
<br><br>

<div id="container_mixer" >
<canvas id="myCanvas" width="640" height="480" style="width:320px; height:240px; position: absolute;" >
</canvas>
</div>

<!-- The OUTPUT (try to plug this stream into Google Meet) -->
<div id="container_output" style="position: absolute; top: 0px; left: 370px;">
<video id="output_vid" width="640" height="480">
<source src="" type="video/mp4">
</video>

<br>
(3) <span id="txt_output"> OUTPUT PREVIEW : </span> <br>

<span id="txt_output_info"> 
- The output stream can be plugged into the Google Meet video tag,<br>
- Or sent to other peers via webRTC / Sockets.<br>
- Or recorded to a video file (using some encoding API like MediaRecorder or WebCodecs)
</span>
<br>
<button id="btn_output_record">Record Output Stream</button>
<br>

</div>


<script>

//# setup buttons for INPUT stage
const btn_input_webcam = document.getElementById("btn_input_webcam");
btn_input_webcam.addEventListener('click',  function (e) { get_input() } );

const btn_input_video = document.getElementById("btn_input_video");
btn_input_video.addEventListener('click',  function (e) { get_input() } );

const btn_input_image = document.getElementById("btn_input_image");
btn_input_image.addEventListener('click',  function (e) { get_input() } );

//# setup buttons for MIXER stage
const btn_fx_pause = document.getElementById("btn_fx_pause");
btn_fx_pause.addEventListener('click',  function (e) { effect_freezeCamera() } );

//////////////////////////////////

//## (1) INPUT : is Media Element
//# access the video tag
const vid = document.getElementById("myVideo");
vid.muted = true;

let txt_input = document.getElementById("txt_input");
let input_cameraStream = null;

let media_width = 0;let media_height = 0;
var stream_output;

////////////////////////////////////////////
//# (1) INPUT : is Media Element
//# access the video tag
const vid_out = document.getElementById("output_vid");
vid_out.muted = true;

vid.onplaying = function() 
{
    //alert("The video is now playing");
    stream_output = vid.captureStream();
    //vid_out.srcObject = stream_output;
    
    vid.requestVideoFrameCallback(updateCanvas);
};

////////////////////////////////////////////
//## (2) MIXER : is Canvas
const canvas_mixer = document.getElementById("myCanvas");
const ctx_mixer = canvas_mixer.getContext('2d');


////////////////////////////////////////////
//## (3) OUTPUT : try sending to Google Meet

var stream_output;

//# set output source (eg: the Canvas)...
stream_output = canvas_mixer.captureStream(25);

/////////////////////////////////////////////////

function effect_freezeCamera()
{
    if( vid.readyState >= 3 )
    {
        if( vid.paused ) { vid.play(); btn_fx_pause.innerHTML = "[ ■ ] Pause Video"; }
        else{ vid.pause(); btn_fx_pause.innerHTML = "[ ► ] Play Video"; }
    }
}

function effect_resumeCamera()
{
    vid.play();
}

function updateCanvas ()
{
    ctx_mixer.drawImage(vid , 0, 0, media_width, 480 );
    vid.requestVideoFrameCallback(updateCanvas);    
}

function get_input()
{
    //alert("### Getting webcam...");
    connect_CameraStream( {video: true} )
    .then
    ( givenStreamObject => 
        {
            input_cameraStream = givenStreamObject;
            let videoTrack = givenStreamObject.getVideoTracks()[0];
            input_cameraStream.addTrack(videoTrack);
            
            txt_input.innerText = "using Camera";
            
            renderVideo();
        }
    )
}

function connect_CameraStream( input_constraints ) 
{
  return navigator.mediaDevices.getUserMedia( input_constraints )
}

function renderVideo() 
{  
    vid.srcObject = input_cameraStream;
 
    vid.onloadedmetadata = function(e) 
    {
        //# update found Width/Height (used by Canvas later on)
        media_width = vid.videoWidth;
        media_height = vid.videoHeight;

        //# show the Camera feed
        vid.play(); 

        vid_out.srcObject = stream_output;
        vid_out.play();

        // update U.I in some way..
        btn_fx_pause.innerHTML = "[ ■ ] Pause Video";
    };
}

</script> 

</body> 
</html>

can you help me integrate it into meets? How would I apply this all into one existing video? — A. Abramov, Apr 16 '23 at 13:25
Thanks for accepting. **(1)** I've been testing Google Meet. I will try to make some extension code for the Answer and update later (again within 24hrs or so). **(2)** Are you asking how to save that (large) output part as one video file such as MP4? The extension code itself should allow you to send anything (camera or image/video file)... I'm working on alternate code for the expected video saving part and that one will be on Github since it'll also be a tutorial about the options to save videos using JS code. I'll let you know when it's ready. — VC.One, Apr 16 '23 at 15:56
I don't mind saving the input, I'm asking how would you go about replacing the original video and it's stream. I'd like to do so in order to send and receive the stream of the frozen image. Thank you! — A. Abramov, Apr 16 '23 at 17:25
My understanding was that Google Meet has a video tag whose input is the webcam and so the site will `captureStream` (to send to others) the video tag with your camera feed. The idea is to hijack that same tag and give it a custom stream so that when Meet does captureStream part it is reading your custom visuals... PS: Sorry for the delay. I forgot the weekend also ends. Interesting topic though so will update when I can. PPS: If you want to send me your extension code to try update it with then it's okay (see email in next comment) or else please wait for my attempt to be ready. — VC.One, Apr 18 '23 at 14:22
I tried to reach via email - do you have an example of how would you go about this? Thank you very much! — A. Abramov, Apr 22 '23 at 20:49

How can I freeze a video MediaStreamTrack?

1 Answers1