8

We record at the conference rooms, where we have multiple speaker conversations. Each speaker has a personal microphone attached and speech from each microphone is recorded as a separate channel(separate audio file).

Our goal is to allow a user to play all the audios(channels) at once. We may have up to 12 separate files. Additionally, we need a user to be able to control the volume of each stream (so just merging files on the backend is not a solution).

What could we use to achieve that? A naive solution to play them all at once is probably not going to work, so we want to know if this kind of challenge was addressed by modern html5 APIs.

--- UPDATE to answer @Amdadan's comment ---

Why not? DId you try it? Are you worried about connection throughput, if you are streaming them? Are you worried about API restrictions?

Yes, we tried. 1) When we call 'play' on all the files - it doesn't start all at the same moment(it starts loading them all, one loads faster, one loads slower, etc). 2) When it's already playing and the file reaches the end of "buffered" time - it also stops while others continue playing.

We tried to use events for handling those cases, but we did not succeed in it - there are a lot of cases to handle.

Are you worried about size, if you load them all up before playing?

Yes, we are. Files can be a couple of hours in length - it can not all be buffered.

Are you worried about connection throughput, if you are streaming them? Are you worried about API restrictions?

No, we use CloudFront connected to s3 - the throughput is ok and there are no specific restrictions

Thank you in advance, - Jack

Jack Juiceson
  • 830
  • 4
  • 12
  • 24
  • Why not? DId you try it? Are you worried about size, if you load them all up before playing? Are you worried about connection throughput, if you are streaming them? Are you worried about API restrictions? – Amadan Jun 04 '19 at 08:35
  • @Amadan - looping over the array of audio files and calling play on them, may produce some delay between 1st one and 12th one, and even if this delay is 100ms it sounds horrible. This is why we are looking for something that allows work on a group of audio files. – Jack Juiceson Jun 04 '19 at 09:57
  • Why would you have 100ms delay? Also, you can manually synchronise your files. For an example, take a look at [this example](https://github.com/mdn/webaudio-examples/tree/master/multi-track): as each file plays, they determine the correct offset to pass to `start` so all the files are in sync (`if (offset == 0) { trackSource.start(); offset = audioCtx.currentTime; } else { trackSource.start(0, audioCtx.currentTime - offset); }`). – Amadan Jun 04 '19 at 14:50
  • Now, in this example, you have pre-loaded tracks, whereas you would be streaming them; if your throughput is not sufficient for the tracks not to lag out and start buffering, then I guess you have to decide what you want to do (should all streams pause? should the buffering stream try to catch up to others and be mute until it manages to do so while all others continue? etc) before asking how to do it :) – Amadan Jun 04 '19 at 15:03
  • @Amadan - It's not working in our case. You used `createBufferSource` in your example, which means that you need to have binary audio data "in memory". It works fine with small audios, but for our files (which can 2 hours long) we need to have a buffering mechanism. Maybe you can suggest how we could use standard buffering easily there? Without it - this solution is not working for us. – Jack Juiceson Jun 04 '19 at 21:04
  • 1
    I did address it in my previous comment - it is rather easy to synchronise the _start_ of several streams (use `Audio`, wait for `canplay` event on all sources, when all are ready start them all); but you need to decide what should happen if one of the sources [stalls out](https://stackoverflow.com/questions/29796420/how-to-identify-that-html5-media-element-is-stalled-and-waiting-for-further-medi)). Another possible approach is, as mentioned by Oliver M Grech, using a multi-track source (where synch is, I believe, done for you automatically). – Amadan Jun 05 '19 at 04:55
  • @Amadan - Yes, I tried it too. Looks like if you handle all the events correctly it should work with just setting currentTime, no need to add the audioContext stuff there. BUT I did not manage to handle that correctly - looks like there are a lot of cases to manage. Example: 1) if one of the sources stalls out 2) there are some tricky things with play/pause promises - when you call play - it's not starting, but it creates a promise and you can not pause it before it's done – Jack Juiceson Jun 05 '19 at 12:08

2 Answers2

1

whatever you opt for make sure there's 0ms delay between the tracks, as this might create phase issues and comb filtering. (for the non audio guys and gals, these issues sounds really bad)

WebM seems to be using Matroska which support multi-track. Details in the link below.

Multi Track Audio (and Video) HTML5

Also check Howler.js, which supports multi-track playback and I think it suits best your needs.

Howler.js on GitHub

Oliver M Grech
  • 3,071
  • 1
  • 21
  • 36
  • I checked the link you posted, the solutions provided there are not supported in Chrome: - audioTracks is not supported by most browsers - looks like only Safari deals with it (https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/audioTracks) - mediagroup would be an amazing solution, which does exactly what I need. But it's also not supported anywhere :( Also, I checked Howler, but looks like it works with mediaBuffers too and for my tests, it tried to download the whole audio to memory. Did I misconfigure it? I did not find in docs anything about buffer sizes... – Jack Juiceson Jun 05 '19 at 12:06
  • Howler should work on all major browsers, check the description on their GITHUB ( Google Chrome 7.0+, Internet Explorer 9.0+, Firefox 4.0+, Safari 5.1.4+, Mobile Safari 6.0+ (after user input), Opera 12.0+, Microsoft Edge ) – Oliver M Grech Jun 05 '19 at 12:46
  • 1
    Also, why is loading it in memory an issue? (if I may ask) Most reputable DAWs and multitrack players load it in memory prior of streaming it out in sync. Personally I think this is normal behavior. How large are your files and number of tracks? – Oliver M Grech Jun 05 '19 at 12:48
  • doesn't work on Chrome note was related to - "Multi Track Audio (and Video) HTML5" suggestion. It's not supported in Chrome. – Jack Juiceson Jun 05 '19 at 13:35
  • 1
    these files are very lengthy! They can be up to 10 hours, so buffering in memory is not an option. – Jack Juiceson Jun 05 '19 at 13:35
  • ouch! I now understand better. I tried my friend :) If you're going to stream that data somehow you have to stream in parallel and play it in sync when the buffers are of equal length.. interesting.. I'll be following – Oliver M Grech Jun 06 '19 at 11:41
  • Last time I checked, Howler.js did not support large streamed files, only locally (buffered) sounds. This makes it unusable for the OT's use case unfortunately. – Marcel Aug 11 '23 at 05:50
1

Maybe using the MediaSync library is an option for you. It allows you to sync two <video/> elements as shown on their page. But that should work with audio as well.

chrisguttandin
  • 7,025
  • 15
  • 21