AVPlayer HLS live stream level meter (Display FFT Data)

Question

I'm using AVPlayer for a radio app using HTTP live streaming. Now I want to implement a level meter for that audio stream. The very best would a level meter showing the different frequencies, but a simple left / right solution would be a great starting point.

I found several examples using AVAudioPlayer. But I cannot find a solution for getting the required informations off AVPlayer.

Can someone think of a solution for my problem?

EDIT I want to create something like this (but nicer)

nice levelmeter

EDIT II

One suggestion was to use MTAudioProcessingTap to get the raw audio data. The examples I could find using the [[[_player currentItem] asset] tracks] array, which is, in my case, an empty array. Another suggestion was to use [[_player currentItem] audioMix] which is null for me.

EDIT III

After years already, there still not seems to be a solution. I did indeed make progress, so I'm sharing it.

During setup, I'm adding a key-value observer to the playerItem:

[[[self player] currentItem] addObserver:self forKeyPath:@"tracks" options:kNilOptions context:NULL];

//////////////////////////////////////////////////////

- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)changecontext:(void *)context
    if ([keyPath isEqualToString:@"tracks"] && [[object tracks] count] > 0) {
        for (AVPlayerItemTrack *itemTrack in [object tracks]) {
            AVAssetTrack *track = [itemTrack assetTrack];

            if ([[track mediaType] isEqualToString:AVMediaTypeAudio]) {
                [self addAudioProcessingTap:track];
                break;
            }
        }
}

- (void)addAudioProcessingTap:(AVAssetTrack *)track {
    MTAudioProcessingTapRef tap;
    MTAudioProcessingTapCallbacks callbacks;
    callbacks.version = kMTAudioProcessingTapCallbacksVersion_0;
    callbacks.clientInfo = (__bridge void *)(self);
    callbacks.init = init;
    callbacks.prepare = prepare;
    callbacks.process = process;
    callbacks.unprepare = unprepare;
    callbacks.finalize = finalise;

    // more tap setup...

    AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];

    AVMutableAudioMixInputParameters *inputParams = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:audioTrack];
    [inputParams setAudioTapProcessor:tap];
    [audioMix setInputParameters:@[inputParams]];

    [[[self player] currentItem] setAudioMix:audioMix];
}

So far so good. This all works, I could find the right track and setup the inputParams and audioMix etc. But unfortunately the only callback, that gets called is the init callback. None of the others will fire at any point.

I tried different (kinds of) stream sources, one of them an official Apple HLS stream: http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8

what is the need of level meters?,if you are playing live streaming, @Julian — Ashish Kakkad, May 27 '14 at 11:25
I'm working on a radio app and my customers designer wanted a level meter for the various frequencies. Like this (but nicer) http://www.oocities.org/henrym24/eq.gif — Julian F. Weinert, May 27 '14 at 12:03
Ok, I have worked on live streaming but not on audio visualization like this, I have posted an answer with some example links may you get help. Happy Coding :) @Julian — Ashish Kakkad, May 27 '14 at 12:46
Have you looked at http://stackoverflow.com/questions/28109516/record-streaming-audio-from-currently-playing-video-in-avplayer? — whyp, Mar 29 '17 at 19:53
Actually not, thanks. I'm assuming that he streams actual audio files. The callbacks aren't called in case of HLS only. I'll have a deeper look later and commened over there. — Julian F. Weinert, Apr 02 '17 at 14:43
Unfortunately not. I revisit this almost once a year, but I think there might be no solution at all :( — Julian F. Weinert, Jul 23 '18 at 12:13

Simon Germain · Answer 1 · 2014-06-02T14:46:38.307

2

Sadly, using an HLS stream with AVFoundation doesn't give you any control over the audio tracks. I ran into the same problem trying to mute an HLS stream, which turned out to be impossible.

The only way you could read audio data would be to tap into the AVAudioSession.

EDIT

You can access the AVAudioSession like this:

[AVAudioSession sharedInstance]

Here's the documentation for AVAudioSession

edited Jun 02 '14 at 14:46

answered Jun 01 '14 at 18:46

Simon Germain

6,834
1
27
42

How do I access the session? – Julian F. Weinert Jun 02 '14 at 12:58
Well, that looks quite promising, thanks. Do then use this session (or any of it's properties) as input for the `MTAudioProcessingTap`? – Julian F. Weinert Jun 03 '14 at 12:41
Good question, but whatever information you need to process audio should be available in there. – Simon Germain Jun 03 '14 at 15:39
@Simon Germain see my comment to Julian re HLS stream. It is indeed possible to insert MTAudioProcessingTap into a HLS Stream hosted via AVPlayer. – MDB983 Jun 06 '14 at 14:58
3

@MDB983 just to recap that two years later... Are you willing to provide us any detail on _how_ it can be done? – Julian F. Weinert Apr 04 '16 at 10:19
Unfortunately not on my side. I was never able to figure out anything despite what's written in my question. Surprisingly undiscussed on the internet... – Julian F. Weinert Nov 04 '16 at 16:16
still need answer how to access meters of avplayer – Yestay Muratov Dec 08 '16 at 16:49
@DavidSeek unfortunately not yet. There's a fairly new comment under the OP, which might be worth looking into – Julian F. Weinert Apr 19 '17 at 16:38
Sorry, dunno why it's actually being downvoted, but so far I wasn't able to achieve my goal by any means... Unfortunately I cannot find the comment @MDB983 was talking about and he also didn't give a hint on his solution so far :/ – Julian F. Weinert Jul 12 '17 at 18:27

score 2 · Answer 2 · answered Sep 04 '17 at 20:46

Measuring audio using AVPlayer looks to be an issue that is still ongoing. That being said, I believe that the solution can be reached by combining AVPlayer with AVAudioRecorder.

While the two classes have seemingly contradictory purposes, there is a work around that allows AVAudioRecorder to access the AVPlayer's audio output.

Player / Recorder

As described in this Stack Overflow Answer, recording the audio of a AVPlayer is possible if you access the audio route change using kAudioSessionProperty_AudioRouteChange.

Notice that the audio recording must be started after accessing the audio route change. Use the linked stack answer as a reference - it includes more details and necessary code.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once you have access to the AVPlayer's audio route and are recording, the measuring is relatively straightforward.

Audio Levels

In my answer to a stack question regarding measuring microphone input I describe the steps necessary to access the audio level measurements. Using AVAudioRecorder to monitor volume changes is more complex than one would think, so I included a GitHub project that acts as a template for monitoring audio changes while recording.

~~~~~~~~~~~~~~~~~~~~~~~~~~ Please Note ~~~~~~~~~~~~~~~~~~~~~~~~~~

This combination during an HLS live stream is not something that I have tested. This answer is strictly theoretical, so it may take a sound understanding of both classes to work out completely.

Wow – that sounds like an obvious workaround, once someone describes it, but hard to think of yourself... Many thanks! I'll have to have look, as the question is more than four years old by now (even though I looked into the topic once in awhile). Please give me a few days to work on that – I'll try to revisit this asap. Again, thanks for sharing your idea! — Julian F. Weinert, Sep 04 '17 at 20:52
Of course. Based on the comments on the answer provided, it seemed to me that there are still plenty of others looking for a viable solution. Not to mention, the number of votes/stars your question has received. — ChrisHaze, Sep 04 '17 at 22:01

AVPlayer HLS live stream level meter (Display FFT Data)

2 Answers2

Player / Recorder

Audio Levels

Linked