Accessing assets from C++ plugin through Flutter

Question

I'm trying to use Google Oboe for a 3D audio processing app due to it's low latency. The app will have a C++ backend, which does the processing, and the frontend is done with Flutter. I'm running a couple of tests to see if it'll work but I'm having issues loading assets from Flutter to Oboe. I checked the example RhythmGame in Oboe's repo, done with Java, but couldn't quiet find a way of doing that straight from Dart to C++. The connection between front and backend is through dart::ffi

Here's what I've tried so far. Based on the example published by Richard Heap here, I changed the noise variable from just a sine wave to a short fragment of a song in a wav file:

class _MyAppState extends State<MyApp> {
  final stream = OboeStream();

  var noise = Float32List(512);
  Timer t;

  @override
  void initState() {
    super.initState();
    // for (var i = 0; i < noise.length; i++) {
    //   noise[i] = sin(8 * pi * i / noise.length);
    // }

    _loadSound();
  }

  void _loadSound() async {
    final ByteData data = await rootBundle.load('assets/song_cut.wav');
    noise = data.buffer.asFloat32List();
  }

(...)

Then this function in Dart calls the Dart wrapper of the native library:

void start() {
    stream.start();
    var interval = (512000 / stream.getSampleRate()).floor() + 1;
    t = Timer.periodic(Duration(milliseconds: interval), (_) {
      stream.write(noise);
     });
  }

The wrapper in Dart is:

void write(Float32List original) {
    var length = original.length;
    var copy = allocate<Float>(count: length)
        ..asTypedList(length).setAll(0, original);

    FfiGoogleOboe()._streamWrite(_nativeInstance, copy, length);
    free(copy);
  }

_streamWrite is the native function in C++:

EXTERNC void stream_write(void* ptr, void* data, int32_t size) {
    auto stream = static_cast<OboeFfiStream*>(ptr);
    auto dataToWrite = static_cast<float*>(data);
    stream->write(dataToWrite, size);
}

void OboeFfiStream::write(float *data, int32_t size) {
    managedStream->write(data, size, 1000000);
}

Now I can hear the song but it comes out with too much distortion. When trying with the sine I could hear it too, but it also had some distortion. I'm not yet using the callback mode in Oboe, since I wanted to try if this worked first.

Answer 1

1 - what format is your WAV file in? Is it 32 bit floats? Don't forget that WAV files have a header, so you should discard the first few tens of bytes (up to the data segment). Be sure that you start reading the audio data on a float boundary (which may not be a multiple of 4 if the header isn't). If necessary, just use a hex editor to ascertain the offset of the float data and start reading there. Or, truncate the header and rename your asset to song_cut.raw. Audacity should be able to produce a header-less raw audio file.

2 - What sample rate is your audio clip recorded at? Does that match the sample rate of the device? (Note that iOS devices are normally 44.1k, but Android devices are frequently 48k. When using an Android emulator on macOS, who knows what the reported sample rate will be! Expect pitch distortion if your rates don't match - or use a resampler. I think Oboe has one. Alternatively, the sample repo associated with the talk contains one you can use.)

3 - note that the timer interval is finely tuned (for demo purposes) to the approximate time taken to deliver 512 samples at the sound card rate. This might be ok for demos, but isn't for real life. Also, your wav file probably doesn't have exactly 512 samples in it. Either adjust your audio loop to 512 samples, or adjust the 512000 constant to match the number of samples in your loop.

4a - You aren't using the callback method yet, but you probably should as soon as possible. One method I've had success with is to use a lock-free circular buffer. The Oboe callback tries to empty the buffer, while the Dart timer routine tries to fill it. The bigger the buffer the less chance there is of an underflow, but the worse the latency.

4b - The ideal solution would be to have the Oboe callback call up into Dart, but I haven't found a way to do that as C->Dart calls must be on the main Dart thread, but the Oboe callbacks are surely on a high-priority IO thread.