7

I am writing program for streaming live audio and video from webcamera to rtmp-server. I work in MacOS X 10.8, so I use AVFoundation framework for obtaining audio and video frames from input devices. This frames come into delegate:

-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer: (CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection ,

where sampleBuffer contains audio or video data.

When I recieve audio data in the sampleBuffer, I'm trying to convert this data into AVFrame and encode AVFramewith libavcodec:

    aframe = avcodec_alloc_frame();  //AVFrame *aframe;
    int got_packet, ret;
    CMItemCount numSamples = CMSampleBufferGetNumSamples(sampleBuffer); //CMSampleBufferRef

    NSUInteger channelIndex = 0;

    CMBlockBufferRef audioBlockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);

    size_t audioBlockBufferOffset = (channelIndex * numSamples * sizeof(SInt16));

    size_t lengthAtOffset = 0;

    size_t totalLength = 0;

    SInt16 *samples = NULL;

    CMBlockBufferGetDataPointer(audioBlockBuffer, audioBlockBufferOffset, &lengthAtOffset, &totalLength, (char **)(&samples));

            const AudioStreamBasicDescription *audioDescription = CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sampleBuffer));

    aframe->nb_samples =(int) numSamples;

    aframe->channels=audioDescription->mChannelsPerFrame;

    aframe->sample_rate=(int)audioDescription->mSampleRate;

     //my webCamera configured to produce 16bit 16kHz LPCM mono, so sample format hardcoded here, and seems to be correct
    avcodec_fill_audio_frame(aframe, aframe->channels, AV_SAMPLE_FMT_S16,

                             (uint8_t *)samples,

                              aframe->nb_samples *

                             av_get_bytes_per_sample(AV_SAMPLE_FMT_S16) *

                             aframe->channels, 0);  
    //encoding audio
    ret = avcodec_encode_audio2(c, &pkt, aframe, &got_packet);
    if (ret < 0) {
        fprintf(stderr, "Error encoding audio frame: %s\n", av_err2str(ret));
        exit(1);
    }

The problem is that when I get so formed frames, I can hear the wanted sound, but it is slowing down and discontinuous (as if after each data frame comes the same frame of silence). It seems that something is wrong in the transformation from CMSampleBuffer to AVFrame , because the preview from the microphone created with AVFoundation from the same sample buffers played normally.

I would be grateful for your help.

UPD: Creating and initializing the AVCodceContext structure 

    audio_codec= avcodec_find_encoder(AV_CODEC_ID_AAC);
    if (!(audio_codec)) {
        fprintf(stderr, "Could not find encoder for '%s'\n",
                avcodec_get_name(AV_CODEC_ID_AAC));
        exit(1);
    }
    audio_st = avformat_new_stream(oc, audio_codec);  //AVFormatContext *oc;
    if (!audio_st) {
        fprintf(stderr, "Could not allocate stream\n");
        exit(1);
    }

      audio_st->id=1;
      audio_st->codec->sample_fmt= AV_SAMPLE_FMT_S16;
      audio_st->codec->bit_rate = 64000;
      audio_st->codec->sample_rate= 16000;
      audio_st->codec->channels=1;
      audio_st->codec->codec_type= AVMEDIA_TYPE_AUDIO;

Aleksei2414904
  • 143
  • 1
  • 2
  • 7
  • Describe how you create and initialize the AVCodceContext structure – pogorskiy May 24 '13 at 10:51
  • For test purposes, can clearly true initialize `AVFrame` structure in accordance with a predetermined format, for example sinusoidal signal, and validate the output. – pogorskiy May 24 '13 at 10:58
  • I created audio frame as presented in ffmpeg example (ffmpeg/doc/samples/muxing.c). With sinusoidal signal, output sounds correct. But when I take audio-samples from microphone instead of sinusoidal samles, problem arises. Also I have found, that if I use 'AV_CODEC_ID_PCM_ALAW' instead of 'AV_CODEC_ID_AAC', output signal sounds slow, but without discontinuities. – Aleksei2414904 May 24 '13 at 22:01
  • Slow signal means that the output samplerate does not match the input. The discontinuity for the AAC codec comes from the fact that the AAC accepts input only **planar** float data (AV_SAMPLE_FMT_FLTP) (each channel is separated from the others) – pogorskiy May 25 '13 at 12:30
  • I tried to initialize the codec with `AV_SAMPLE_FMT_FLTP`, but received error _[libfaac @ 0x104121c00] Specified sample format fltp is invalid or not supported Could not open audio codec: Invalid argument_ during performing `avcodec_open2`. I also checked for all other sample formats, and found that the codec successfully initialized only if I use the `AV_SAMPLE_FMT_P16` or `AV_SAMPLE_FMT _S16P`. But in both cases the problem with discontinuity remains. – Aleksei2414904 May 26 '13 at 10:40
  • Both of these formats are planar – pogorskiy May 26 '13 at 18:57
  • I have tried different configurations for AAC codec, but it still not work. I found in [https://github.com/libav/libav/blob/master/libavcodec/libfaac.c] where looks like this codec support only `AV_SAMPLE_FMT_P16`. Also I added resampler before AAC encoding, because flv container supports only 11025 22050 and 44100 samplerates, but it did not help. I checked the sample rate, sample format and bit rate. It turns out that the problem may be only the number of channels or channel layout. Is AAC codec support mono mode? Or need only stereo? – Aleksei2414904 Jun 01 '13 at 15:24

0 Answers0