FFMPEG Reading audio from memory doesn't work

Question

When I try to instantiate this struct, my program crashes:

struct MemoryAVFormat {
    MemoryAVFormat(const MemoryAVFormat &) = delete;

    AVFormatContext *ctx;
    AVIOContext *ioCtx;

    MemoryAVFormat(char *audio, size_t audio_length) :
            ctx(avformat_alloc_context()),
            ioCtx(create_audio_buffer_io_context(audio, audio_length)) {

        if (ctx == nullptr)
            throw audio_processing_exception("Failed to allocate context");

        if (ioCtx == nullptr)
            throw audio_processing_exception("Failed to allocate IO context for audio buffer");

        ctx->pb = ioCtx;
        ctx->flags |= AVFMT_FLAG_CUSTOM_IO;

        int err = avformat_open_input(&ctx, "nullptr", NULL, NULL);
        if (err != 0)
            throwAvError("Error configuring context from audio buffer", err);
    }

    AVIOContext *create_audio_buffer_io_context(char *audio, size_t audio_length) const {
        return avio_alloc_context(reinterpret_cast<unsigned char *>(audio),
                                  audio_length,
                                  0,
                                  audio,
                                  [](void *, uint8_t *, int buf_size) { return buf_size; },
                                  NULL,
                                  NULL);
    }

    ~MemoryAVFormat() {
        av_free(ioCtx);
        avformat_close_input(&ctx);
    }
}

I've read and tried every single tutorial on doing this and none of them work

Has anyone got this working before?

crashes on the line: int err = avformat_open_input(&ctx, "nullptr", NULL, NULL);

"Trying to instantiate this struct fails:" - what do you mean ? Does it give errors at compile time ? at runtime ? — TheEagle, Feb 02 '21 at 03:48
I'll try and show the stack trace, but its all machine code - I don't understand it — Tobi Akinyemi, Feb 02 '21 at 03:49
Oof, i do not know how to understand this, sorry .... But i think it might have to do something with passing `nullptr` as arguments in `int err = avformat_open_input(&ctx, nullptr, nullptr, nullptr);` ... — TheEagle, Feb 02 '21 at 03:55
@HTNW look at the tutorials. people either pass null filename or a "DUMMYNAME". The API gets the audio from the `ctx->pb`. Hence the flag: `AVFMT_FLAG_CUSTOM_IO` — Tobi Akinyemi, Feb 02 '21 at 04:26
@HTNW If you look at the documentation, there's even an old example doing this - but didn't work. All the posts on SO don't work — Tobi Akinyemi, Feb 02 '21 at 04:27

gkv311 · Answer 1 · 2021-02-02T12:11:16.480

avio_alloc_context() documentation specifies that buffer parameter should be allocated by av_malloc() and moreover it will be released by AVIOContext destructor and can be reallocated at any time:

 * @param buffer Memory block for input/output operations via AVIOContext.
 *        The buffer must be allocated with av_malloc() and friends.
 *        It may be freed and replaced with a new buffer by libavformat.
 *        AVIOContext.buffer holds the buffer currently in use,
 *        which must be later freed with av_free().

In your code sample you omit details of allocation of audio buffer, but I suppose that it doesn't meet these requirements, so that crash happens when FFmpeg tries to release or reallocate audio buffer.

I guess passing an entire audio file content as an externally allocated buffer wouldn't work with AVIOContext - this API is really meant to be used with a temporary buffer for streaming data from somewhere else (file, web or another memory buffer).

I don't have a complete sample to see if it will work as expected, but code might look like this (you will probably need to tune read() function and consider implementing seeking procedure as well):

struct MemoryAVFormat {
    MemoryAVFormat(const MemoryAVFormat &) = delete;

    AVFormatContext *ctx;
    AVIOContext *ioCtx;

    char *audio;
    size_t audio_length;
    size_t audio_offset;

    MemoryAVFormat(char *theAudio, size_t theAudioLength)
    : ctx(avformat_alloc_context()),
      ioCtx(nullptr),
      audio(theAudio),
      audio_length(theAudioLength),
      audio_offset(0) {
        ioCtx = create_audio_buffer_io_context();
        if (ctx == nullptr)
            throw audio_processing_exception("Failed to allocate context");

        if (ioCtx == nullptr)
            throw audio_processing_exception("Failed to allocate IO context for audio buffer");

        ctx->pb = ioCtx;
        ctx->flags |= AVFMT_FLAG_CUSTOM_IO;

        int err = avformat_open_input(&ctx, "nullptr", NULL, NULL);
        if (err != 0)
            throwAvError("Error configuring context from audio buffer", err);
    }

    int read (uint8_t* theBuf, int theBufSize) {
        int aNbRead = std::min (int(audio_length - audio_offset), theBufSize);
        if(aNbRead == 0) { return AVERROR_EOF; }
        memcpy(theBuf, audio + audio_offset, aNbRead);
        audio_offset += aNbRead;
        return aNbRead;
    }

    int64_t seek(int64_t offset, int whence) {
         if (whence == AVSEEK_SIZE) { return audio_length; }
         audio_offset = offset;

         if(audio == NULL || audio_length == 0) { return -1; }
         if     (whence == SEEK_SET) { audio_offset = offset; }
         else if(whence == SEEK_CUR) { audio_offset += offset; }
         else if(whence == SEEK_END) { audio_offset = audio_length + offset; }

         //if(audio_offset < 0) { audio_offset  = 0; } else
         //if(audio_offset > audio_length) { audio_offset = audio_length; }
         return offset;
    }

    AVIOContext *create_audio_buffer_io_context() {
        const int aBufferSize = 4096;
        unsigned char* aBufferIO = (unsigned char* )av_malloc(aBufferSize + AV_INPUT_BUFFER_PADDING_SIZE);
        return avio_alloc_context(aBufferIO,
                                  aBufferSize,
                                  0,
                                  this,
                                  [](void* opaque, uint8_t* buf, int bufSize)
                                  { return ((MemoryAVFormat* )opaque)->read(buf, bufSize); },
                                  NULL,
                                  [](void* opaque, int64_t offset, int whence)
                                  { return ((MemoryAVFormat* )opaque)->seek(offset, whence); });
    }

    ~MemoryAVFormat() {
        av_free(ioCtx);
        avformat_close_input(&ctx);
    }
}

An alternative to implementing AVIOContext interface and using avformat_open_input() could be passing an audio buffer as a payload of a custom AVPacket directly to decoder, if you know in advance in which audio format your stream is (e.g. by skipping creation of AVFormatContext at all). I did this for decoding image pixmaps, but don't know if it could be (easily) applied to audio.

Im getting an error for overallocating an array for the audio_stream (I thought because the AV read was corrupted) but in reality only the `AVStream::duration` property was corrupted. What I don't understand is why this property is only corrupted when using customIO? And can I obtain the stream length in another way? — Tobi Akinyemi, Feb 02 '21 at 09:49
Maybe this is not because it is a custom IO, but because it is unseekable? — gkv311, Feb 02 '21 at 10:02
Is that what you were saying with: `(you will probably need to tune read() function and consider implementing seeking procedure as well)`? — Tobi Akinyemi, Feb 02 '21 at 11:11
Implemented `inline auto read(uint8_t *theBuf, int theBufSize) { int read_count = std::min(int(audio_length - audio_offset), theBufSize); if (read_count == 0) { return AVERROR_EOF; } std::memcpy(theBuf, audio + audio_offset, read_count); audio_offset += read_count; return read_count; } inline int64_t seek(int64_t offset, int whence) { if (whence == AVSEEK_SIZE) return audio_length; audio_offset = offset; return offset; }` and it worked. Thank you very much, I looked everywhere for this stuff! — Tobi Akinyemi, Feb 02 '21 at 11:28
Yes, that's what I've meant. I've put seek() function to the answer for completeness. — gkv311, Feb 02 '21 at 12:14

FFMPEG Reading audio from memory doesn't work

1 Answers1