0

I am successfully recording audio, storing it, and reloading it into Audio object for playback in the browser.

However, I would like to be able to record new audio and "splice" it into the original recording at a certain time offset, completely replacing that portion of the original recording onward from that offset. For instance, suppose I recorded 10 seconds of audio from the microphone, and subsequently, I wished to "record over" the last 5 seconds of that audio with 8 seconds of totally new audio, ending up with a new bufferArray that I could persist. I've spent some hours researching this and still am pretty vague on how to do it. If anybody has any suggestions, I'd be appreciative.

The closest examples I could find involved getting two buffers and attempting to concatenate them, as in this fiddle. However, I'm having some difficulties relating this fiddle to my code and what I need to do. Certain posts mention I need to know the sample rate, create new buffers with that same sample rate, and copy data from two buffers into the new buffer. But there are precious few working examples on this technique, and I'm heavily scratching my head trying to figure this out from just the MDN docs.

Here's the code I have working now.

const saveRecordedAudio = (e) => {
  console.log("Audio data available", e.data);
  const reader = new FileReader();
  reader.addEventListener("loadend", function() {
    // reader.result contains the contents of blob as a typed array
    let bufferArray = reader.result;
    // From: https://stackoverflow.com/questions/9267899/arraybuffer-to-base64-encoded-string
    let base64String = btoa([].reduce.call(new Uint8Array(bufferArray),function(p,c){return p+String.fromCharCode(c)},''));
    // persist base64-encoded audio data on filesystem here.
    storeRecordedAudio(base64String); // defined elsewhere and not shown here for brevity's sake
  });
  reader.readAsArrayBuffer(e.data);

  const audioUrl = window.URL.createObjectURL(e.data);
  // Put the recorded audio data into the browser for playback.
  // From: http://stackoverflow.com/questions/33755524/how-to-load-audio-completely-before-playing (first answer)
  const audioObj = new Audio (audioUrl);
  audioObj.load();

};

const recordAudio = () => {
  navigator.getUserMedia = ( navigator.getUserMedia ||
                             navigator.webkitGetUserMedia ||
                             navigator.mozGetUserMedia ||
                             navigator.msGetUserMedia);
  if (navigator.getUserMedia) {
    navigator.getUserMedia (
      { // constraints - only audio needed for this app
        audio: true
      },
      // Success callback
      function(stream) {
        const mediaRecorder = new MediaRecorder(stream);
        mediaRecorder.ondataavailable = audio.saveRecordedAudio;
      }),
      // fail callback
      function(err) {
        console.log('Could not record.');
      }
   );
  }
};


// I've already fetched a previously recorded audio buffer encoded as
// a base64 string, so place it into an Audio object for playback
const setRecordedAudio => (b64string) => {
  const labeledAudio = 'data:video/webm;base64,' + b64String;
  const audioObj = new Audio(labeledAudio);
  audioObj.load();
};
dhilt
  • 18,707
  • 8
  • 70
  • 85
Will Kessler
  • 565
  • 1
  • 7
  • 17
  • just saving the bufferArrays passed to a number of Blob constructors may help. If you have the orig. corresponding arrays , you may add, subtract them from an aggregating bufferArray and pass that last BA to a new blob constructor, thereby realizing the reqmt for mix & match of audio clips. – Robert Rowntree Nov 24 '18 at 15:55
  • https://github.com/higuma/mp3-lame-encoder-js/blob/master/src/post.js#L42 - mp3 example of where you might adjust the raw , arrayBuffer in order to mix, aggregate respective clips and their orig. arrayBuffrs. – Robert Rowntree Nov 24 '18 at 16:04
  • 1
    https://www.npmjs.com/package/blob-to-buffer to get back to arrayBuff from a blob. – Robert Rowntree Nov 24 '18 at 19:46
  • Thank you for the tips. This may help me indeed. However, I'm still not sure if I can just copy an arbitrary part of the first bufferArray into a new array, then aggregate with another bufferArray (using Blob), since I don't know how to relate number of bytes to copy, to milliseconds of recorded sound. Ie, suppose I need 5.27s of the first clip to be combined with the entirety of the second clip. How would I calculate how many bytes to copy out of the first bufferArray? – Will Kessler Nov 25 '18 at 05:47
  • if you work on the basis of clips, this approach is feasible ,but at the sub-clip or sub-blob level i dont think it is so great when, as u say you have to infer based on timestamps the portions of clip related buffers that u want to work with. – Robert Rowntree Nov 25 '18 at 08:15
  • http://w3c.github.io/media-source/#idl-def-appendmode - re: how messy the sub-buffer stuff can be. – Robert Rowntree Nov 25 '18 at 08:25
  • one idea i just had is to use the `start()` method as outlined in @cryptobird's answer and store just the byte lengths of received chunks (at say 100ms granularity), create a blob from the chunks, and then use `blob-to-buffer` to go back to a bufferArray. Then later, I could just add up these lengths to calculate how many bytes to copy from the first clip's bufferArray to include a subsection of it in a new edited clip, with a 0.1s resolution. – Will Kessler Nov 25 '18 at 16:22
  • 1
    i would also look at ffmpeg on node back end to manage the timestamps for you. you would only do CLI or API calls and ffmpeg does all the timestamp stuff 4 u. downside is all the clips have to be on the cloud/back end for CLI or API to work. – Robert Rowntree Nov 25 '18 at 17:28
  • Good tip but... this is for a client-side only app, there is no back end (jupyter notebook extension). There may be some python libs I could use inside jupyter but ideally i'd avoid that if the front-end can do all the work. – Will Kessler Nov 25 '18 at 18:24
  • This seems relevant to your suggestion about ffmpeg, but client-side: https://github.com/muaz-khan/Ffmpeg.js/ – Will Kessler Nov 25 '18 at 18:54
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/184225/discussion-between-will-kessler-and-robert-rowntree). – Will Kessler Nov 25 '18 at 18:55

1 Answers1

-1

If I get what you're trying to do correctely, here is my way to deal with this problem:

1- We should declare a blob array and store data in it:

var chuncks = [];

2- We need to know the number of seconds recorded :

We should use the start(1000) method with the timeslice parameter in order to limit the number of milliseconds in each blob, in this case, we know that every chunk of data is approximately 1 second long ( chunks.length = number of seconds recorded).And to fill in the chunks array with data, ondataavailable event handler is our friend :

mediaRecorder.ondataavailable = function( event ) {
    chunks.push( event.data );
}

Now, we have to deal with simple Arrays instead of Buffer Arrays :-D (In your example, we should delete or override the last 5 items and keep adding blobs to it)

3- And finally, when our recording is ready (chunks array):

we should join all together using the Blob constructor:

new Blob(chunks, {'type' : 'audio/webm;codecs=opus'});

I hope this helps :-D

CryptoBird
  • 508
  • 1
  • 5
  • 22
  • Thank you, this is a useful tip. I will study the start() method. But as I commented above, what I really need is to be able to preserve an arbitrary number of milliseconds of the first clip and combine it with the entirety of the second clip. With your suggestion, wouldn't I be limited to a resolution of 1s? I mean, I suppose the chunks could be 100ms long which may be small enough to not cause dropout... but it could get unwieldy to keep all the chunks around forever just in case the user wants to add another clip. – Will Kessler Nov 25 '18 at 05:49
  • The above **start(1000)** is just an example, you could use 100 instead of 1000 if that works for you. – CryptoBird Nov 26 '18 at 14:41
  • As far as I know, this is the only easy way to do that, and please, keep in mind that, It's not easy to crop out an exact specific part of an audio file ( 5.27s as you mentioned in your comment above), cause, it contains multiple sound signals. – CryptoBird Nov 26 '18 at 14:44
  • Please, let me know if there is any other easy way to do that. Thanks. – CryptoBird Nov 26 '18 at 14:45
  • This is just plain wrong for a couple of reasons. First, the timeslice provided to `start` is just a *suggestion* to the browser on how frequently it should get chunks of data. There is no guarantee that they will be timed precisely, and if you run it for long enough or use sub-1000ms values it quickly becomes out of sync with the elapsed time. The second is that the Blob data is not uniformly distributed, so you cannot just cut and paste it however you like, as you can easily end up removing header information and making the file corrupted. – lawrence-witt Nov 27 '20 at 19:15