Why aren't Safari or Firefox able to process audio data from MediaElementSource?

Question

Neither Safari or Firefox are able to process audio data from a MediaElementSource using the Web Audio API.

var audioContext, audioProcess, audioSource,
    result = document.createElement('h3'),
    output = document.createElement('span'),
    mp3 = '//www.jonathancoulton.com/wp-content/uploads/encodes/Smoking_Monkey/mp3/09_First_of_May_mp3_3a69021.mp3',
    ogg = '//upload.wikimedia.org/wikipedia/en/4/45/ACDC_-_Back_In_Black-sample.ogg',
    gotData = false, data, audio = new Audio();
 
function connect() {
  audioContext = window.AudioContext ? new AudioContext() : new webkitAudioContext(),
  audioSource  = audioContext.createMediaElementSource( audio ),
  audioScript  = audioContext.createScriptProcessor( 2048 );
 
  audioSource.connect( audioScript );
  audioSource.connect( audioContext.destination );
  audioScript.connect( audioContext.destination );
  audioScript.addEventListener('audioprocess', function(e){
    if ((data = e.inputBuffer.getChannelData(0)[0]*3)) {
      output.innerHTML = Math.abs(data).toFixed(3);
      if (!gotData) gotData = true;
    }
  }, false);
}
 
(function setup(){
  audio.volume = 1/3;
  audio.controls = true;
  audio.autoplay = true;
  audio.src = audio.canPlayType('audio/mpeg') ? mp3 : ogg;
  audio.addEventListener('canplay', connect);
  result.innerHTML = 'Channel Data: ';
  output.innerHTML = '0.000';
  document.body.appendChild(result).appendChild(output);
  document.body.appendChild(audio);
})();

Are there any plans to patch this in the near future? Or is there some work-around that would still provide the audio controls to the user?

For Apple, this something that could be fixed in the WebKit Nightlies or will we have to wait until Safari 8.0 release to get HTML5 <audio> playing nicely with the Web Audio API? The Web Audio API has existed in Safari since at least version 6.0 and I initially posted this question long before Safari 7.0 was released. Is there a reason this wasn't fixed already? Will it ever be fixed?

For Mozilla, I know you're still in the process of switching over from the old Audio Data API, but is this a known issue with your Web Audio implementation and is it going to be fixed before the next release of Firefox?

That's what `audioElement.readyState < 3 ? audioElement.addEventListener('canplay', connect) : connect();` is there for. And while iOS suffers from this same *bug*, I'm specifically curious about the desktop version. — idbehold, Dec 19 '12 at 17:58
Could you give more detail on how it "fails"? Is there an error message produced, or does the "onaudioprocess" callback never run? — Matt Diamond, Dec 19 '12 at 22:10
@MattDiamond I linked to the test case in the post: http://jsfiddle.net/2cd73/ The audio plays, but the `ScriptProcessor` doesn't receive anything but 0 values. — idbehold, Oct 30 '13 at 23:04

score 5 · Accepted Answer · edited May 23 '17 at 12:02

This answer is quoted almost exactly from my answer to a related question: Firefox 25 and AudioContext createJavaScriptNote not a function

Firefox does support `MediaElementSource` if the media adheres to the Same-Origin Policy, however there is no error produced by Firefox when attempting to use media from a remote origin.

The specification is not really specific about it ^{(pun intended)}, but I've been told that this is an intended behavior, and the issue is actually with Chrome… It's the Blink implementations (Chrome, Opera) that need to be updated to require CORS.

MediaElementSource Node and Cross-Origin Media Resources:

From: Robert O'Callahan <robert@ocallahan.org>
Date: Tue, 23 Jul 2013 16:30:00 +1200
To: "public-audio@w3.org" <public-audio@w3.org>

HTML media elements can play media resources from any origin. When an element plays a media resource from an origin different from the page's origin, we must prevent page script from being able to read the contents of the media (e.g. extract video frames or audio samples). In particular we should prevent ScriptProcessorNodes from getting access to the media's audio samples. We should also information about samples leaking in other ways (e.g. timing channel attacks). Currently the Web Audio spec says nothing about this.

I think we should solve this by preventing any non-same-origin data from entering Web Audio. That will minimize the attack surface and the impact on Web Audio.

My proposal is to make MediaElementAudioSourceNode convert data coming from a non-same origin stream to silence.

If this proposal makes it into spec it will be nearly impossible for a developer to even realize why his MediaElementSource is not working. As it stands right now, calling createMediaElementSource() on an <audio> element in Firefox 26 actually stops the <audio> controls from working at all and throws no errors.

What dangerous things can you do with the audio/video data from a remote origin? The general idea is that without applying the Same-Origin Policy to a MediaElementSource node, some malicious javascript could access media that only the user should have access to (session, vpn, local server, network drives) and send its contents—or some representation of it—to an attacker.

The HTML5 media elements don't have these restrictions by default. You can include remote media across all browsers by using the <audio>, <img>, or <video> elements. It's only when you want to manipulate or extract the data from these remote resources that the Same-Origin Policy comes into play.

[It's] for the same reason that you cannot dump image data cross-origin via <canvas>: media may contain sensitive information and therefore allowing rogue sites to dump and re-route content is a security issue. - @nmaier

So are we still screwed on Safari for processing Audio Data? — Cabbibo, Jul 17 '14 at 06:56
There is a Firefox bug that appears to prevent this from working even when it should: [https://bugzilla.mozilla.org/show_bug.cgi?id=937718](https://bugzilla.mozilla.org/show_bug.cgi?id=937718) — aldel, Nov 25 '14 at 17:18
The example jsfiddle (http://jsfiddle.net/9eGSp/1/) now seems to be working (playing audio) in Firefox 34. Maybe this bug got fixed? Anyone know? — Jordan Eldredge, Dec 13 '14 at 23:38

score 1 · Answer 2 · answered May 08 '15 at 18:51

1

createMediaElementSource() does not work properly in Safari 8.0.5 (and possibly earlier) but is fixed in Webkit Nightly as of 10600.5.17, r183978

answered May 08 '15 at 18:51

Ian

592
5
21

Why aren't Safari or Firefox able to process audio data from MediaElementSource?

2 Answers2

Firefox does support `MediaElementSource` if the media adheres to the Same-Origin Policy, however there is no error produced by Firefox when attempting to use media from a remote origin.

MediaElementSource Node and Cross-Origin Media Resources:

Linked

Why aren't Safari or Firefox able to process audio data from MediaElementSource?

2 Answers2

Firefox does support MediaElementSource if the media adheres to the Same-Origin Policy, however there is no error produced by Firefox when attempting to use media from a remote origin.

MediaElementSource Node and Cross-Origin Media Resources:

Linked

Firefox does support `MediaElementSource` if the media adheres to the Same-Origin Policy, however there is no error produced by Firefox when attempting to use media from a remote origin.