0

I'm new to web audio API, I used a demo and managed to separate the two channels of PC audio input source and drive a double demo analyzer view. I've realized that most functions work by handling automatically the whole stream of audio data, but my question is this:

how can I get a single sample from each channel of the stereo input source when I want it (programmatically) and put it on a variable, then another one at a different time?

var input = audioContext.createMediaStreamSource(stream);
var options = {
    numberOfOutputs : 2 
}
var splitter = new ChannelSplitterNode(audioContext, options);
input.connect( splitter );
splitter.connect(analyser1, 0, 0);
splitter.connect(analyser2, 1, 0);
dllb
  • 59
  • 1
  • 7

1 Answers1

0

If you're not too concerned about latency, the MediaRecorder interface can be used to capture raw audio data from a streaming input source (like the one returned by AudioContext.createMediaStreamSource). Use the dataavailable event or MediaRecorder.requestData method to access the raw data as a Blob. There's a fairly straightforward example of this in samdutton/simpl on GitHub. (Also there's a related Q&A on StackOverflow here.)

More generally if you can get the audio you want to analyze into a AudioBuffer, the AudioBuffer.getChannelData method can be used to extract the raw PCM sample data associated with an audio channel.

However if you're doing this in "real-time" - i.e., if you're trying to process the live input audio for "live" playback or visualization - then you'll probably want to look at the AudioWorklet API. Specifically, you'll want to create an AudioWorkletProcessor that examines the individual samples as part of the process handler.

E.g., something like this:

// For this example we'll compute the average level (PCM value) for the frames
// in the audio sample we're processing.

class ExampleAudioWorkletProcessor extends AudioWorkletProcessor {

  process (inputs, outputs, parameters) {
    // the running total
    sum = 0

    // grab samples from the first channel of the first input connected to this node
    pcmData = inputs[0][0]

    // process each individual sample (frame)
    for (i = 0; i < pcmData.length; i++ ) {
      sum += pcmData[i]
    }

    // write something to the log just to show it's working at all
    console.log("AVG:", (sum / pcmData.length).toFixed(5))

    // be sure to return `true` to keep the worklet running
    return true
  }

}

but that's neither bullet-proof nor particularly efficient code. (Notably you don't really want to use console.log here. You'll probably either (a) write something to the outputs array to send audio data to the next AudioNode in the chain or (b) use postMessage to send data back to the main (non-audio) thread.)

Note that since the AudioWorkletProcessor is executing within the "audio thread" rather than the main "UI thread" like the rest of your code, there are some hoops you must jump through to set up the worklet and communicate with the primary execution context. That's probably outside the scope of what's reasonable to describe here, but here is a complete example from MDN, and there are a large number of tutorials that can walk you through the steps if you search for keywords like "AudioWorkletProcessor", "AudioWorkletNode" or just "AudioWorklet". Here's one example.

Rod
  • 2,226
  • 22
  • 21
  • Thanks, but the devil is in the details. I need to put a sample to a variable (exactly when I want it) and send it for general purposes to the main thread (eg for visualization), not for the sole purpose of sending it to the speakers (I managed that already). I wonder if I can do that via postMessage, but the authors always assume postMessage is used for other purposes -the ones they have imagined, so there are no proper examples -the sin of assumption! And the audio api is a mess as usual... – dllb Sep 14 '21 at 15:29
  • 1
    FWIW I *often* use postMessage more or less the way you describe (i.e., I compute some value in "real-time" based on audio data within the worklet then send periodically update the main thread via postMessage). E.g. that average level example I gave wasn't totally artificial. That's not the actual computation you should use but I do something very similar to that report the "volume" of the audio stream. Note that you probably don't want to call postMessage *every* time process executes (because it happens frequently) I usually apply some kind of postMessage-every-n-process-calls logic. – Rod Sep 14 '21 at 16:02
  • 1
    More generally I think the structure of the Worklet API pretty much demands you follow a pattern of: 1. Use the `output` parameter to the process method if you want to do something to the audio output and 2. Use postMessage for literally everything else (anything that's not audio data you want to send downstream to the rest of the AudioNode chain). If you have a value based on the audio data that you want to share with the main thread I don't think you have a lot of other (non-hacky) options *other* than sending it via postMessage. – Rod Sep 14 '21 at 16:06
  • 1
    BTW regarding "I need to put a sample to a variable (exactly when I want it)" note there are couple of ways of triggering that (or communicating to your worklet which specific sample you want to grab): One way is to use the port to post a message from your main thread to your worklet, which would sorta work "on demand" but you should account for some latency (i.e., if you want the sample precisely at timestamp T you'll need to post that message a little before time T so the message doesn't arrive too late, it's not gonna be instantanous). [...] – Rod Sep 14 '21 at 16:17
  • 1
    [...] another, more complicated but potentially interesting approach would be to use some kind of custom a-rate based AudioParameter (see https://developer.mozilla.org/en-US/docs/Web/API/AudioParam#a-rate). These are the data in the third argument to the process method, which in the general case might be something like the gain or whatever. The a-rate params allow you to specify a value for each frame in sample, so you could do something like "when myCustomParam[i] = 1 that's the precise sample I want to share with the main thread" – Rod Sep 14 '21 at 16:21
  • 1
    [...] that last option is probably the most precise (and at some level, WebAudio-native implementation) but also requires more fiddling. If it's sufficient for your use case it might be enough to just post a message to your worklet that says "send me the data point closest to audio context time T" and work that out with a little bit of math in your process method. – Rod Sep 14 '21 at 16:25
  • Very helpful info, I'll try these, thanks! – dllb Sep 14 '21 at 18:01