7

How could an audio track be extracted from a video in HTML5 Javascript as the raw audio data? I.e. an array of samples?

I am completely new to the HTML5 Video API so an example would be great.

James
  • 30,496
  • 19
  • 86
  • 113
  • I don't know if Javascript is suitable for such tasks. Please refer to this topic: http://stackoverflow.com/questions/11182587/extract-audio-from-video-stream-using-javascript – Rafael Eyng Jul 08 '13 at 16:46

1 Answers1

8

The Web Audio API is exactly what you want. In particular, you want to feed a MediaElementAudioSourceNode into an AnalyserNode. Unfortunately, the Web Audio API is only implemented in Chrome (somewhat implemented in FF), and even Chrome doesn't have full support for MediaElementAudioSourceNode yet.

var context = new webkitAudioContext();

// feed video into a MediaElementSourceNode, and feed that into AnalyserNode
// due to a bug in Chrome, this must run after onload
var videoElement = document.querySelector('myVideo');
var mediaSourceNode = context.createMediaElementSource(videoElement);
var analyserNode = context.createAnalyser();
mediaSourceNode.connect(analyserNode);
analyserNode.connect(context.destination);

videoElement.play();

// run this part on loop to sample the current audio position
sample = new Float32Array(analyser.frequencyBinCount);
analyser.getFloatFrequencyData(sample);
apsillers
  • 112,806
  • 17
  • 235
  • 239
  • Thanks very much, this was just what I'm after. Hopefully more browsers will support this soon. – James Jul 08 '13 at 17:23
  • Could we go the other way too? Given some samples and footage, could a video be created? – James Jul 08 '13 at 17:39
  • @James Maybe? The Audio API might allow you to buffer samples and re-play them somehow, but I'm not sure. In any case, you couldn't make an actual video *file*, but you might synchronize a muted video with audio playback. – apsillers Jul 08 '13 at 18:09
  • Firefox will support this API very soon. Most of it is targeting Firefox 24 (in about 12 weeks). It should reach the beta channel in about 5 weeks. MediaElementAudioSourceNode is not yet implemented, but should be available on Nightly in the next few days or weeks. A dev is actively implementing this right now. More docs: [Web Audio API on Firefox](https://hacks.mozilla.org/2013/07/web-audio-api-comes-to-firefox/ ) and [MDN Web Audio API doc](https://developer.mozilla.org/en-US/docs/Web_Audio_API ) – teoli Jul 11 '13 at 08:06
  • @apsillers Do we have a way to achieve this without scanning the whole video and concating the Float32Arrays every second? – starkm Mar 07 '18 at 10:24
  • 1
    @starkm I think you might be looking for [Is there a way to use the Web Audio API to sample audio faster than real-time?](https://stackoverflow.com/questions/8074152/is-there-a-way-to-use-the-web-audio-api-to-sample-audio-faster-than-real-time) which briefly describes the faster-than-real-time `OfflineAudioContext` (see also [MDN's documentation on it](https://developer.mozilla.org/en-US/docs/Web/API/OfflineAudioContext). I've never used it, though. – apsillers Mar 07 '18 at 12:15