4

I'm working on a library that can extract audio from HTML5 <audio> and <video> elements and send it to a remote server for speech-to-text processing. (e.g. automatic subtitles.) When I test manually, everything works as expected, but when karma runs my test, it always fails for <audio> elements.

For context, HTML5 audio script processors immediately get a stream of audio as soon as they are connected to an <audio> element, but it's all 0's (no sound) until the element begins playing.

When my tests run in karma, it appears that the element never begins playing; I just get 0's even though I've called .play() on the element. Also, I don't hear any audio coming out of my speakers, which should happen at the same time.

(I get that audio during tests could be super-annoying and probably should be disabled by default, but is there a way to enable it for cases like mine?)

If you're interested in seeing the code, it's at https://github.com/watson-developer-cloud/speech-javascript-sdk but it's still a bit of a mess, and is currently configured to skip the tests that would fail. If necessary, I'll try and set up a stripped down example, but I'm hoping I won't need to.

I'm on OS X if it makes a difference. I haven't tried this on Windows or Linux yet.

Mr Lister
  • 45,515
  • 15
  • 108
  • 150
Nathan Friedly
  • 7,837
  • 3
  • 42
  • 59

2 Answers2

2

Found it, at least for firefox. <audio> elements apparently have stronger CORS rules than I realized! I was setting a Access-Control-Allow-Origin: * header on my file server, because I knew it was on a different port then where Karma was running my tests, but apparently that's not enough.

I can't seem to find any documentation on this anywhere, but I happened to notice a chrome Error about CORS and did a little more digging & experimentation. It seems that scriptProcessors for cross-domain <audio> elements need the the same crossOrigin attribute that <img> tags need to use their content in a <canvas>.

So, in the end, going from 

  var audioElement = new Audio();
  audioElement.src = "http://localhost:9877/audio.wav";

to

  var audioElement = new Audio();
  audioElement.crossOrigin = "anonymous";
  audioElement.src = "http://localhost:9877/audio.wav";

Solved the issue for me in firefox, and chrome works sometimes but seems to have some other issues (perhaps related to load time?).

I believe the second issue in chrome is that I was calling .play() right away and then ending my test when it emitted an ended event.. but that event seems to also fire when the buffer runs empty, which I think is a bug. I changed it to wait on the canplaythrough event before calling .play() and it seems to be reliably passing the test now.

Nathan Friedly
  • 7,837
  • 3
  • 42
  • 59
0

Some browser implementations will block attempts to automatically play <audio> elements unless a user interaction has occurred (for obvious reasons).

There are several suggestions in this similar question, but they are all attempting to circumvent a designed feature and would probably not be very reliable across multiple browsers / devices.

Depending on your setup and which browsers you are targeting, you might need to factor in a 'click to start' button in your test.

Community
  • 1
  • 1
duncanhall
  • 11,035
  • 5
  • 54
  • 86