69

After sweating blood and tears I've finally managed to set up a Node C++ addon and shove a web-platform standard MediaStream object into one of its C++ methods for good. For compatibility across different V8 and Node.js versions, I'm using Native Abstractions for Node.js (nan):

addon.cc

NAN_METHOD(SetStream)
{
    Nan::HandleScope scope;
    v8::Local<v8::Object> mediaStream = info[0]->ToObject();
}

addon.js

setStream(new MediaStream());

For what it's worth, this works correctly (i.e. it does not obliterate the renderer process on sight), and I can verify the presence of the MediaStream object, e.g. by returning its constructor name from the C++ method:

addon.cc

info.GetReturnValue().Set(mediaStream->GetConstructorName());

When called from JavaScript through setStream, this would return the string MediaStream, so the object is definitely there. I can also return the mediaStream object itself and everything will work correctly, so it's indeed the object I need.

So, how would I read audio data (i.e. audio samples) from this MediaStream object in C++? As a sidenote, the actual data read (and processing) would be done in a separate std::thread.

Bounty Update

I understand this would be sort of easier/possible if I were compiling Electron and/or Chromium myself, but I'd rather not get involved in that maintenance hell.

I was wondering if it would be possible without doing that, and as far as my research goes, I'm convinced I need 2 things to get this done:

  1. The relevant header files, for which I believe blink public should be adequate
  2. A chromium/blink library file (?), to resolve external symbols, similarly to the node.dylib file

Also, as I said, I believe I could compile chromium/blink myself and then I would have this lib file, but that would be a maintenance hell with Electron. With this in mind, I believe this question ultimately comes down to a C++ linking question. Is there any other approach to do what I'm looking for?

Edit

ScriptProcessorNode is not an option in my case, as its performance makes it nearly unusable in production. This would require to process audio samples on the ui/main thread, which is absolutely insane.

Edit 2

AudioWorklets have been available in Electron for some time now, which, unlike the ScriptProcessorNode (or worse, the AnalyzerNode), is low-latency and very reliable for true C++ backed audio processing even in real time.

If someone wants to go ahead and write an AudioWorklet-based answer, I'll gladly accept, but beware: it's a very advanced field and a very deep rabbit hole, with countless obstacles to get through even before a very simple, general-purpose pass-through prototype (especially so because currently in Electron, Atomics-synced, buffered cross-thread audio processing is required to pull this off because https://github.com/electron/electron/issues/22503 -- although getting a native C++ addon into one audio renderer thread, let alone multiple threads at the same time, is probably equally as challenging).

John Weisz
  • 30,137
  • 13
  • 89
  • 132
  • 3
    I'm not getting your question. but u can check these two : https://github.com/common-tater/wkwebview-getusermedia-shim/blob/master/WKWebViewGetUserMediaShim/WKWebViewGetUserMediaShim.m ... http://realtimeweekly.co/how-to-use-webrtc-in-your-ios-projects-part-ii/ – nullqube Mar 09 '18 at 06:04
  • @nullqube I've been checking out WebRTC for this task, but it seems it's [not possible](https://stackoverflow.com/questions/48791065/uncompressed-unencrypted-unaltered-raw-transfer-of-real-time-pcm-audio-data-t) to stream raw, uncompressed audio. If it was possible, I could simply start a new Electron window process, stream audio there, do ScriptProcessorNode-based processing in that separate window process, and stream back. But there's always significant amounts of lossy compression _hardcoded_ into WebRTC, as well as very high latency. – John Weisz Mar 11 '18 at 09:31
  • does this help? https://discuss.atom.io/t/multiple-streaming-recording-in-media-recorder/54239 – nullqube Aug 12 '18 at 06:49
  • Did you try using `WebWorker` on the JavaScript side to transfer your work into "background thread" then call the C++ part from it?? – user9335240 Oct 14 '18 at 20:43
  • Have you tried WebAssemby – Clonk Dec 21 '18 at 16:14
  • I'm not an expert in node bindings, but from what I found in existing addons, did you try something like: ```v8::Local method = Nan::Get(obj, Nan::New("getTracks").ToLocalChecked()).ToLocalChecked(); Nan::MaybeLocal result = Nan::CallAsFunction(method->ToObject(), obj , 0, NULL);``` to call js methods of objects? – Wlodzislav K. Jan 31 '19 at 08:47
  • @WlodzislavK. The problem is that accessing MediaStream samples is not a JS exposed as a JS method, because it's not part of the web API. – John Weisz Mar 29 '19 at 15:12
  • 3
    Its not possible by doing it just with the V8 API. If you're capable of knowing the Blink engine version of your electron or chromium version, then you could add a AudioDestinationConsumer to the source of one of the audio tracks on the media stream. https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/mediastream/media_stream_source.h?sq=package:chromium&g=0&l=123 I've currently not yet tested that, but In a few months I will and post a more detailed answer :) – WolverinDEV Apr 06 '19 at 14:21
  • I think you can't do that from node, you need a browser context, to use the analyzer node of the web audio api. This is a browser native api: https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode/getFloatFrequencyData . What you can do is set a local server from your scripts and send the datas from the browser to that server – NVRM May 02 '19 at 17:13
  • You say it comes to down a linker issue / maintenance hell - this is not obvious to me - can you go into more details as to why ? – darune Aug 30 '19 at 11:00

1 Answers1

1

The MediaStream header is part of Blink's renderer modules, and it's not obvious to me how you could retrieve this from nan plugin.

So, instead let's look at what you do have, namely a v8::Object. I believe that v8::Object exposes all the functionality you need, it has:

  • GetPropertyNames()
  • Get(context, index)
  • Set(context, key, value)
  • Has(context, key)

Unless you really need a strictly defined interface, why not avoid the issue altogether and just use the dynamic type that you already have?

For getting audio data out specifically, you would need to call getAudioTracks() on the v8::Object, which probably looks something like this?

Note: I don't think you need a context, v8 seems to be happy with it being empty: v8/src/api/api.cc

Should look something like this, plus some massaging of types in and out of v8.


v8::MaybeLocal<v8::Value> get_audio_tracks = mediaStream->Get("getAudioTracks");
// Maybe needs to be v8::Object or array?
if (!get_audio_tracks.IsEmpty()) {
    v8::Local<v8::Value> audio_tracks = get_audio_tracks.ToLocalChecked()();
}
jaypb
  • 1,544
  • 10
  • 23