expo-av recording to flask api

Question

I am trying to build an app using React-Native that will record a sound on a mobile device, using expo-av, and send it to a Flask API.

The problem is that the data I receive from expo-av, into the API, is in an unexpected format. I would like to retrieve the signal/wave data of the sound but when I plot the data to inspect it I get plot-A below, instead of something more like plot-B.

This resource was very useful and contains a fully reproducible example: https://www.tderflinger.com/en/react-native-audio-recording-flask

How can I retrieve in Python (flask) the sample's data that would could help me produce a plot like these? And what does plot-A represent?

plot-A:

plot-B

Python code for plot-A:

import numpy as np
import matplotlib.pyplot as plt

# This is a preview of the data from expo-av
# I made the flask api print(request.get_data()) and copied it from the terminal  
string = b'\x00\x00\x00\x18ftyp3gp4\x00\x00\ ... x00\x00\x00('

# binary to np array
data = np.frombuffer(string, dtype=np.uint8)

plt.plot(data)
plt.show()

Thank you

Claudio Paladini · Accepted Answer · 2023-02-09T10:50:53.940

The solution I found was the following.

Recording settings on expo-av:

  async function startRecording() {

    try {
      await Audio.requestPermissionsAsync();
      await Audio.setAudioModeAsync({
        allowsRecordingIOS: true,
        playsInSilentModeIOS: true,
      });
      const { recording } = await Audio.Recording.createAsync(

        Audio.RecordingOptionsPresets.HIGH_QUALITY

      );

      console.log('Starting recording.. ');

      setRecording(recording);
      console.log('Recording started');

    } catch (err) {
      console.error("Failed to start recording", err);
    }
  }

Sending recording to Flask:

  async function postAudioData() {

    try {
      const response = await FileSystem.uploadAsync(
        FLASK_BACKEND,
        recordingPath
      );
      const body = JSON.parse(response.body);
      console.log(body)
    } catch (err) {
      console.error(err);
    }

  }

Writing recording POST-ed to Flask as .ma4:

@app.route('/audio', methods=['POST'])
def predict():

    data = request.get_data()
    # data = np.frombuffer(data, dtype=np.uint8)

    with open("file.m4a","wb") as file:
        file.write(data)

Converting .m4a to .wav:

from pydub import AudioSegment

AudioSegment.converter = "C:/ffmpeg/bin/ffmpeg.exe"
AudioSegment.ffmpeg = "C:/ffmpeg/bin/ffmpeg.exe"
AudioSegment.ffprobe ="C:/ffmpeg/bin/ffprobe.exe"

m4a_file = 'file.m4a'
wav_filename = r"file.wav"
track = AudioSegment.from_file(m4a_file,  format= 'm4a')
file_handle = track.export(wav_filename, format='wav')

Importing .wav:

import wave

with wave.open(path,'r') as wav:
        signal = np.frombuffer(
            wav.readframes(-1),
            dtype='int16'
        )
        frame_rate = wav.getframerate()

I think the issue of not retrieving the expected data resulted from trying to extract the signal/wave data directly from the binary data received from expo-av. Plot-A is probably the result of charting directly that binary representation of the sound file, instead of going through the steps above and retrieving the actual data of the sound's signal.

More details can be found here: https://www.tderflinger.com/en/react-native-audio-recording-flask

Happy to hear about better ways of doing this!

expo-av recording to flask api

1 Answers1