Dealing With Raw Audio JavaScript

Question

The goal of my project is to be able to manipulate the raw audio data, like in this question: Play raw audio with JavaScript. There are five steps:

• Record the audio
• Convert the resulting blob into an ArrayBuffer
• Manipulate the ArrayBuffer of audio data with a DataView (for example, make it twice as fast or twice as loud).
• Convert back into a blob
• Play the manipulated audio

When I convert the blob of audio data to an ArrayBuffer, and the ArrayBuffer to an Int8Array so that I can view the data, it looks like this (I am only showing the first few values, not the whole thing):

[26,69,-33,-93,-97,66,-122,-127,1,66,-9,-127,1,66,-14,-127,4,66,-13,-127,8,66,-126,-124,119,101,98,109,66,-121,-127,4,66,-123,-127,2,24,83,-128,103,1,-1,-1,-1,-1,-1,-1,-1,21,73,-87,102,-103,42,-41,-79,-125,15,66,64,77,-128,-122,67,104,114,111,109,101,87,65,-122,67,104,114,111,109,101,22,84,-82,107,-65,-82,-67,-41,-127,1,115,-59,-121,-103,-103,-9,91,-16,59,77,-125,-127,2,-122,-122,65,95,79,80,85,83,99,-94,-109,79,112,117,115,72,101,97,100,1,1,0,0,-128,-69,0,0,0,0,0,-31,-115,-75,-124,71,59,-128,0,-97,-127,1,98,100,-127,32,31,67,-74,117,1,-1,-1,-1,-1,-1,-1,-1,-25,-127,0,-93,65,52,-127,0,0,-128,-5,-125,2,-60,-1,-2,127,-4,102,-43,102,31,98,87,-112,-55,-17,-20,-88,89,39,108,97,84,44,38,-113,61,36,122,7,39,61,27,-103,-91,-23,80,64,36,9,-4,-51,-127,12,109,38,100,99,-101,-18,74,124,108,59,71,81,23,-30,93...]

Since it doesn't seem possible to go from an Int8Array back to an ArrayBuffer, and eventually back to a Blob which can be played as audio, I decided to use DataView to manipulate the ArrayBuffer.

When I don't change the ArrayBuffer at all, it can successfully convert it back into a blob and back into audio. However, when I try to change the ArrayBuffer with a DataView, I get the following error:

index.html:1 
Uncaught (in promise) DOMException: Failed to load because no supported source was found.

After looking online, the error seems to be some sort of CORS issue, but that doesn't make sense because it works when I don't change the ArrayBuffer. Does the issue have to do with the data type or something?

Here is my entire JavaScript:

let AudioContext = window.AudioContext || window.webkitAudioContext;
var mediaRecorder;

function start(){ //start recording
    audio = navigator.mediaDevices.getUserMedia({
        audio: true,
        video: false
    }).then(record);
}

function record(stream) {
    mediaRecorder = new MediaRecorder(stream);
    mediaRecorder.start(); //start recording

    mediaRecorder.addEventListener("dataavailable", event => {
        event.data.arrayBuffer().then(processAudio); //event.data is a blob, converts it into an arrayBuffer
    });
}

function processAudio(buffer){
    var view = new DataView(buffer); //to change the ArrayBuffer
    var array = new Int8Array(buffer); //this typed array contains all of the audio data
    
    view.setInt16(0, 1); //when I comment out this line and don't change the ArrayBuffer at all, it works.

    var newBlob = new Blob([buffer]); //convert back into blob

    var newAud = document.createElement("audio"); //create audio element to play the recording
    newAud.src = URL.createObjectURL(newBlob);
    newAud.play();
    document.body.appendChild(newAud);
}

function done(){ mediaRecorder.stop(); }

I would also like to mention that when I try to graph the Int8Array, I get random static like the image below:

I am pretty sure that it is supposed 8-bit encoding because the byte length of the ArrayBuffer is not always a mutliple of two.

Any help is appreciated.

Answer 1

I think you left out a key step in your list: converting the incoming data to meaningful PCM values.

It looks like you have been delivered a stream of signed bytes. That the values all lie between -127 and 127 is suggestive.

The actual PCM most likely consists of values that are assembled with either two or three or maybe even four bytes per PCM value. The most common audio format is 16-bit encoding. Two bytes will need to be concatenated in either little-endian or big-endian order in order to get your PCM values.

Once you have your signal represented in PCM, you can manipulate it meaningfully. Note that you aren't restricted to just halving or doubling: it's also possible to extrapolate intermediate points in the signal with linear interpolation, which gives you the opportunity to play back the sound at any speed you desire.

Of course, after your manipulations to the signal, the individual PCM values will have to be broken back down into the expected byte order format that your system requires. However, not knowing the specifics of JavaScript audio, perhaps there are methods or functions that allow you to stream PCM without doing the byte-level conversions.

My experience is with Java, not JavaScript, but this looks like a very similar situation. I do know that Android, for example, allows one to directly stream PCM data as, for example, signed normalized floats.

Answer 2

For audio, the unsigned Uint8Array is needed as opposed to the signed Int8Array. As Phil answered below, you'll have raw PCM audio bytes. <audio> cannot play raw PCM bytes, but it can play other formats like WAV. Or you can play the PCM bytes directly with the WebAudio API. See answers to both at: How to Play RAW Audio Files?