WebRTC: Switch camera

Question

I want to be able to switch the camera while in the middle of a conversation with WebRTC, without renegotiating the call.

Suppose we have 2 "mediaSources" objects using the MediaStreamTrack.getSources method:

{
    id: "id_source_1" | "id_source_2",
    facing: "user" | "environment",
    kind: "kind_1" | "kind_2",
    label: "label_1" | "label_2"
 }

We start the call with "source_1" (facing the "user"), and we want the user to be able to switch to "source_2" (facing the "environment").

In my current code when the user clicks the "switch camera" button the following is executed: (callingSession is the current WebRTC session)

var mediaParams = {
       audio: true,
       video: { deviceId : source_2.id},
       options: {
           muted: true,
           mirror: true
       },
       elemId: 'localVideo'
};

callingSession.getUserMedia(mediaParams, function (error, stream) {
if (error) {
   console.error('error getting user media');
} else {
          var oldVideoTracks = callingSession.localStream.getVideoTracks();      
          var newVideoTracks = stream.getVideoTracks();

          if (oldVideoTracks.length > 0 && newVideoTracks.length > 0) {
               callingSession.localStream.removeTrack(oldVideoTracks[0]);
               callingSession.localStream.addTrack(newVideoTracks[0]);
          } 
       }
 });

As you can see the mediaParams constraints is now set to the "source_2", we pass this mediaParams with new constraints to the getUserMedia method. Then we get the video tracks from both the old and the new stream.

The main problem in this code, is that the old stream is still exactly the same as the new stream, even with the new constraints passed to the getUserMedia method, so obviously the same video tracks, and of course nothing happens, and the camera is not switched !!!

Am I doing something wrong in this code ? is there any way to switch the camera without without renegotiating the call in WebRTC ? what about the experimental method applyConstraint() I am not able to see it in chrome ?

Thank you.

UPDATE My WebRTC app is an ionic app with crosswalk => the webview is chrome

Answer 1

At the time of writing this post, the WebRTC specification is very promising but still the implementation of this specification varies from browser to another. Currently the Chrome implementation is still old. Nevertheless thanks to jib comments and to this SO answer and also more understanding of the SDP (Session Description Protocol) I can now switch the camera using Chrome.

First the constraints to my getUserMedia method were wrong, here is how I managed to pass the right constraints :

var mediaParams = {
            // the other constraints
            video: {mandatory: {sourceId: source_2.id}}
            // ...
        };

After calling getUserMedia with the mediaParams argument we need to remove the current stream from the peer connection then add the new one like this :

peerConnection.removeStream(peerConnection.getLocalStreams()[0]);
peerConnection.addLocalStream(stream);

Those two lines of code will trigger the onnegotiationneeded on the peerConnection object, means peer 1 must tell peer 2 he changed the stream so he needs a new description. Thats why we need to create an offer, set the new description and send this new description to the peer :

peerConnection.createOffer()
.then(function (offer) {
      peerConnection.setLocalDescription(offer);
})
.then(function () {
      send(JSON.stringify({ "sdp": peerConnection.localDescription }));
});

At this point its up to you how you want to send the SDP. (In my use case I had to send them using WebSockets.)

Once the other peer receives the new SDP he must set it in his own peer connection :

var obj = JSON.parse(answer).sdp;
peerConnection.setRemoteDescription(new RTCSessionDescription(obj));

I hope this will help someone someday.

Answer 2

In order to replace the current camera, without losing the candidate process you must use the following example code, obviously replacing your own variables:

navigator.mediaDevices.getUserMedia(constraints)
.then(function(stream){
    localVideo.srcObject = stream;
    stream.getVideoTracks().forEach(function(track) {
        var sender = peerConnCallee.getSenders().find(function(s) {
          return s.track.kind == track.kind;
        });
        sender.replaceTrack(track);
    });
})
.catch(function(e) { });

You can see it working in the next site: Can Peek

Answer 3

In case you need to switch camera and update the video track for both remote peer connection and your local video stream (e.g. you're show the local video to the user himself), here is a more comprehensive solution. It uses TypeScript, but can be easily converted into plain JS if needed. The solution seems long, but it's well structured and split into multiple re-usable functions.

// assuming RTCPeerConnection is initialized in variable `peerConnection`,
// and local MediaStream is stored in variable `localStream` 
// (you can also extract local stream from the video element's "srcObject")

enum CameraFacingMode {
    FRONT = 'user',
    BACK = 'environment'
};

let facingMode: CameraFacingMode.FRONT; // assuming this was the initial value
 
async function toggleFacingMode() {
    facingMode = facingMode === CameraFacingMode.FRONT ? CameraFacingMode.BACK : CameraFacingMode.FRONT;
    return switchCameraFacingMode(localStream, peerConnection, facingMode);
}


/**
 * Switch to the new facingMode without stopping the existing stream and hopefully without re-negotiating RTC connection
 */
async function switchCameraFacingMode(localStream: MediaStream, peerConnection: RTCPeerConnection, facingMode: CameraFacingMode) {
    const oldVideoTrack = localStream.getVideoTracks()[0];
    const oldVideoConstraints = oldVideoTrack.getConstraints();
    const newConstraints: MediaStreamConstraints = {
        audio: false,
        video: {
            ...oldVideoConstraints,
            facingMode
        }
    };

    // must stop the old video track before creating a new stream, 
    // otherwise some devices throw an error because we're trying
    // to access 2 cameras at the same time
    oldVideoTrack.stop();

    const bufferStream = await navigator.mediaDevices.getUserMedia(newConstraints);
    const newVideoTrack = bufferStream.getVideoTracks()[0];

    await replaceVideoTrackInPeerConnection(peerConnection, newVideoTrack);

    replaceVideoTrackInLocalStream(localStream, newVideoTrack);
}

function replaceVideoTrackInLocalStream(localStream: MediaStream, track: MediaStreamTrack): void {
    const oldVideoTrack = localStream.getVideoTracks()[0];

    localStream.removeTrack(oldVideoTrack);
    localStream.addTrack(track);
}

function replaceVideoTrackInPeerConnection(connection: RTCPeerConnection, track: MediaStreamTrack): Promise<void> {
    const sender = connection.getSenders().find(
        s => s.track.kind === track.kind
    );

    return sender?.replaceTrack(track);
}

Special attention to the functions replaceVideoTrackInLocalStream and replaceVideoTrackInPeerConnection as they show how to replace video track in remote connection as well as local video stream.

Answer 4

replaceTrack API was defined just for this.

Eventually chrome will support the RTCRtpSender.replaceTrack method (http://w3c.github.io/webrtc-pc/#rtcrtpsender-interface), which can be used to replace a track without renegotiation. You can track the development of that feature in Chrome here: https://www.chromestatus.com/feature/5347809238712320

It's already available to some extent in the native API. But it's currently under development, so use at your own risk.

Answer 5

No need to re-negotiate when we can replaceTrack

Following worked for me for multiple peerConnections.

https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/replaceTrack

'''

navigator.mediaDevices
  .getUserMedia({
    video: {
      deviceId: {
        exact: window.selectedCamera
      }
    }
  })
  .then(function(stream) {
    let videoTrack = stream.getVideoTracks()[0];
    PCs.forEach(function(pc) {
      var sender = pc.getSenders().find(function(s) {
        return s.track.kind == videoTrack.kind;
      });
      console.log('found sender:', sender);
      sender.replaceTrack(videoTrack);
    });
  })
  .catch(function(err) {
    console.error('Error happens:', err);
  });

'''

Answer 6

I don't know if it's only me but if someone else cannot call navigator.mediaDevices.getUserMedia() because your camera is busy (notReadableError: Could not start video source), try adding this before calling getUserMedia()

for(let p in pc){
    let pName = pc[p];
    pc[pName] && pc[pName].getSenders().forEach(s => s.track && s.track.stop());
}
myStream.getTracks().length ? myStream.getTracks().forEach(track => track.stop()) : '';

It might be busy by the remote peers as well as your local stream.