Flask send pyaudio to browser

Question

I'm sending my servers microphone's audio to the browser (mostly like this post but with some modified options).

All works fine, until you head over to a mobile or safari, where it doesn't work at all. I've tried using something like howler to take care of the frontend but with not success (still works in chrome and on the computer but not on the phones Safari/Chrome/etc). <audio> ... </audio> works fine in chrome but only on the computer.

function play_audio() {
  var sound = new Howl({
    src: ['audio_feed'],
    format: ['wav'],
    html5: true,
    autoplay: true
  });
  sound.play();
}

How does one send a wav-generated audio feed which is 'live' that works in any browser?

EDIT 230203:

I have narrowed the error down to headers (at least what I think is causing the errors).

What headers should one use to make the sound available in all browsers?

Take this simple app.py for example:

from flask import Flask, Response, render_template
import pyaudio
import time

app = Flask(__name__)

@app.route('/')
def index():
    return render_template('index.html', headers={'Content-Type': 'text/html'})

def generate_wav_header(sampleRate, bitsPerSample, channels):
    datasize = 2000*10**6
    o = bytes("RIFF",'ascii')
    o += (datasize + 36).to_bytes(4,'little')
    o += bytes("WAVE",'ascii')
    o += bytes("fmt ",'ascii')
    o += (16).to_bytes(4,'little')
    o += (1).to_bytes(2,'little')
    o += (channels).to_bytes(2,'little')
    o += (sampleRate).to_bytes(4,'little')
    o += (sampleRate * channels * bitsPerSample // 8).to_bytes(4,'little')
    o += (channels * bitsPerSample // 8).to_bytes(2,'little')
    o += (bitsPerSample).to_bytes(2,'little')
    o += bytes("data",'ascii')
    o += (datasize).to_bytes(4,'little')
    return o

def get_sound(InputAudio):

    FORMAT = pyaudio.paInt16
    CHANNELS = 2
    CHUNK = 1024
    SAMPLE_RATE = 44100
    BITS_PER_SAMPLE = 16

    wav_header = generate_wav_header(SAMPLE_RATE, BITS_PER_SAMPLE, CHANNELS)

    stream = InputAudio.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=SAMPLE_RATE,
        input=True,
        input_device_index=1,
        frames_per_buffer=CHUNK
    )

    first_run = True
    while True:
       if first_run:
           data = wav_header + stream.read(CHUNK)
           first_run = False
       else:
           data = stream.read(CHUNK)
       yield(data)


@app.route('/audio_feed')
def audio_feed():

    return Response(
        get_sound(pyaudio.PyAudio()),
        content_type = 'audio/wav',
    )

if __name__ == '__main__':
    app.run(debug=True)

With a index.html looking like this:

<html>
  <head>
    <title>Test audio</title>
  </head>
  <body>
    <button onclick="play_audio()">
      Play audio
    </button>
    <div id="audio-feed"></div>
  </body>
<script>

  function play_audio() {
    var audio_div = document.getElementById('audio-feed');
    const audio_url = "{{ url_for('audio_feed') }}"
    audio_div.innerHTML = "<audio controls><source src="+audio_url+" type='audio/x-wav;codec=pcm'></audio>";
  }

</script>
</html>

Fire upp the flask development server python app.py and test with chrome, if you have a microphone you will hear the input sound (headphones preferably, otherwise you'll get a sound loop). Firefox works fine too.

But If you try the same app with any browser on an iPhone you'll get no sound, and the same goes for safari on MacOS.

There's no errors and you can see that the byte stream of the audio is getting downloaded in safari, but still no sound.

What is causing this? I think I should use some kind of headers in the audio_feed response but with hours of debugging I cannot seem to find anything for this.

EDIT 230309:

@Markus is pointing out to follow RFC7233 HTTP Range Request. And that's probably it. While firefox, chrome and probably more browsers on desktop send byte=0- as header request, safari and browsers used on iOS send byte=0-1 as header request.

Did you check support and browser versions https://caniuse.com/audio ? — Gonzalo Odiard, Feb 03 '23 at 12:00
Anything here is related? https://stackoverflow.com/questions/8689758/audio-element-not-working-properly-with-safari — Gonzalo Odiard, Feb 03 '23 at 15:42
can you try with different format, like mp4, also can you look at the permissions of the safary/chrome/etc. apps to see if they can use the microphone? — Tony, Feb 09 '23 at 13:33
The browser has nothing to do with the microphone itself. That’s server only. — destinychoice, Feb 09 '23 at 14:58
Can confirm that this works on browser (windows Firefox) and fails on mobile (iPhone Safari & Firefox). Very strange. — flakes, Mar 09 '23 at 05:12
Your example works out of the box for me on Chrome for Android. Do you get the audio on the route `/audio_feed`? Are you able to play any wave files on Safari? — etuardu, Mar 10 '23 at 23:18
@etuardu yes, that’s the point and issue, iOS (all browsers) and safari is not working! — destinychoice, Mar 11 '23 at 11:43
In the question you stated that it does not work "on the phones", so I thought the issue would affect Android as well. I recommend you to correct that statement and also add the `iOS` tag. — etuardu, Mar 11 '23 at 15:05

Markus · Accepted Answer · 2023-03-12T19:32:02.583

EDITED 2023-03-12

It turns out that it is sufficient to convert the audio live stream to mp3. For this you can use ffmpeg. The executable has to be available in the execution path of the server process. Here is a working draft tested with windows laptop as server and Safari on iPad as client:

from subprocess import Popen, PIPE
from threading import Thread
from flask import Flask, Response, render_template
import pyaudio

FORMAT = pyaudio.paFloat32
CHANNELS = 1
CHUNK_SIZE = 4096
SAMPLE_RATE = 44100
BITS_PER_SAMPLE = 16

app = Flask(__name__)


@app.route('/')
def index():
    return render_template('index.html', headers={'Content-Type': 'text/html'})


def read_audio(inp, audio):
    while True:
        inp.write(audio.read(num_frames=CHUNK_SIZE))


def response():
    a = pyaudio.PyAudio().open(
        format=FORMAT,
        channels=CHANNELS,
        rate=SAMPLE_RATE,
        input=True,
        input_device_index=1,
        frames_per_buffer=CHUNK_SIZE
    )

    c = f'ffmpeg -f f32le -acodec pcm_f32le -ar {SAMPLE_RATE} -ac {CHANNELS} -i pipe: -f mp3 pipe:'
    p = Popen(c.split(), stdin=PIPE, stdout=PIPE)
    Thread(target=read_audio, args=(p.stdin, a), daemon=True).start()

    while True:
        yield p.stdout.readline()


@app.route('/audio_feed', methods=['GET'])
def audio_feed():
    return Response(
        response(),
        headers={
            # NOTE: Ensure stream is not cached.
            'Cache-Control': 'no-cache, no-store, must-revalidate',
            'Pragma': 'no-cache',
            'Expires': '0',
        },
        mimetype='audio/mpeg')


if __name__ == "__main__":
    app.run(host='0.0.0.0')

In index.html change the type to audio/mp3:

<!DOCTYPE html>
<html>
  <head>
    <title>Test audio</title>
  </head>
  <body>
    <button onclick="play_audio()">
      Play audio
    </button>
    <div id="audio-feed"></div>
  </body>
<script>
  function play_audio() {
    var audio_div = document.getElementById('audio-feed');
    const audio_url = "{{ url_for('audio_feed') }}"
    audio_div.innerHTML = "<audio preload='all' controls><source src=" + audio_url + " type='audio/mp3'></audio>";
  }
</script>
</html>

Disclaimer: This is just a basic demo. It opens an audio-ffmpeg subprocess for each call to the audio_feed handler. It doesn't cache data for multiple requests, it doesn't remove unused threads and it doesn't delete data that isn't consumed.

Credits: how to convert wav to mp3 in live using python?

It's really getting to my head this one. Returning 206 breaks Chrome, and still nothing in Safari. Really odd. — destinychoice, Feb 10 '23 at 15:59
@destinychoice Do you want to show your code how you have tried? You can add it to this question or even better create a new one and share the link here. Perhaps it‘s better to start with streaming a wav file or a generated sound stream, e.g. streaming silence or a sine signal. This way it is easier for others to reproduce your issue. — Markus, Feb 10 '23 at 23:43
The code in the question is reproducible if you have a microphone. — destinychoice, Mar 07 '23 at 21:04
@destinychoice The code posted in the question doesn't implement RFC7233, yet. I'm sorry to hear that my answer didn't help you to solve this. — Markus, Mar 08 '23 at 07:38
The links in your answer are implementing that on files that already exists. This might be the way, but I’m uncertain on how to do this when the actual file size is unknown. — destinychoice, Mar 08 '23 at 15:16
This isn't bad at all, works great on iOS and Safari desktop (and the other ones too). I have something to work with here! However, I cannot figure out what the `read_audio` exactly does? — destinychoice, Mar 13 '23 at 20:24
In this example `read_audio` is used as background thread that continuesly copies chunks of audio data from `pyaudio` into the ffmpeg pipe. This way the response routine that reads the output from the ffmpeg pipe is decoupled from this task. Otherwise waiting for output from the ffmpeg pipe could block copying data from pyaudio into the pipe. I hope this explanation helps?! — Markus, Mar 13 '23 at 21:55

Flask send pyaudio to browser

1 Answers1

Linked