What is a good set of constraints for lowest latency audio playback/monitoring with the MediaStream Recording API?

Question

I'm currently spiking out a music application with HTML5/JS and am attempting to achieve the lowest latency I can with the MediaStream Recording API. The app allows a user to record music with a camera and microphone. While the camera and microphone are on, the code will allow the user to hear and see themselves.

At the moment I have:

  const stream = await navigator.mediaDevices.getUserMedia(
    {
      video: true,
      audio: {
        latency: {exact: 0.003},
      }
    }
  );

  // monitor video and audio (i.e. show it to the user)
  this.video.srcObject = stream;
  this.video.play();

If I go any lower on the latency requirement, I get an OverConstrained error. The latency is okay (better than the default) but still not great for the purposes of hearing yourself while you're recording. There is a slight, perceptible lag from when you strum a guitar and hear it in your headphones.

Are there other optimizations here I can make to achieve better results? I don't care about the quality of the video and audio as much, so maybe lowering resolution, sample rates, etc. could help here?

In addition to the processing latency through JavaScript, it's also important to consider the time it takes to digitize incoming audio from the microphone and physically move that data. There will never be zero latency, especially if you're using WASAPI instead of something like ASIO (this is not something you can control from the browser). — IronFlare, Sep 03 '19 at 13:22

Mosè Raguzzini · Accepted Answer · 2019-08-30T12:40:34.323

latency of 0.003 is a very, very low latency (3ms) and not noticeable by human beings ear.

Said that, the latency cannot be 0, when we talk of digital audio. Although you set a very low value, it is not guaranteed that the latency actually match for various reason, in case the system can't match the latency the promise will be rejected.

As you can read here in docs:

Constraints which are specified using any or all of max, min, or exact are always treated as mandatory. If any constraint which uses one or more of those can't be met when calling applyConstraints(), the promise will be rejected.

Notice: different browsers and differents OS behave differently.

Chrome

Chrome, in some canary build introduced a low latency feature called Live Web Audio Input:

// success callback when requesting audio input stream
function gotStream(stream) {
    window.AudioContext = window.AudioContext || window.webkitAudioContext;
    var audioContext = new AudioContext();

    // Create an AudioNode from the stream.
    var mediaStreamSource = audioContext.createMediaStreamSource( stream );

    // Connect it to the destination to hear yourself (or any other node for processing!)
    mediaStreamSource.connect( audioContext.destination );
}

navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia;
navigator.getUserMedia( {audio:true}, gotStream );

Here you can see in action some demos taking advantage of that feature:

Yea maybe this is a limitation of the time it takes to process the data and render it back out to the user. In this example, https://mdn.mozillademos.org/en-US/docs/Web/API/Media_Streams_API/Constraints$samples/Example_Constraint_exerciser?revision=1493751, although the latency is really low (shouldn't be perceptible by the human ear), you can clap and hear that the latency there isn't at 3ms — Tomek, Sep 03 '19 at 16:20
I wonder if the way to go here is to use the Media Stream Recording API just for video, and use the Web Audio API's Live Web Audio Input to playback the audio to the user for input monitoring purposes. — Tomek, Sep 03 '19 at 16:36

What is a good set of constraints for lowest latency audio playback/monitoring with the MediaStream Recording API?

1 Answers1