Clipping sound with opus on Android, sent from IOS

Question

I am recording audio in IOS from audioUnit, encoding the bytes with opus and sending it via UDP to android side. The problem is that the sound is playing a bit clipped. I have also tested the sound by sending the Raw data from IOS to Android and it plays perfect.

My AudioSession code is

      try audioSession.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker])
        try audioSession.setPreferredIOBufferDuration(0.02)
        try audioSession.setActive(true)

My recording callBack code is:

func performRecording(
    _ ioActionFlags: UnsafeMutablePointer<AudioUnitRenderActionFlags>,
    inTimeStamp: UnsafePointer<AudioTimeStamp>,
    inBufNumber: UInt32,
    inNumberFrames: UInt32,
    ioData: UnsafeMutablePointer<AudioBufferList>) -> OSStatus
 {
var err: OSStatus = noErr

err = AudioUnitRender(audioUnit!, ioActionFlags, inTimeStamp, 1, inNumberFrames, ioData)

if let mData = ioData[0].mBuffers.mData {
    let ptrData = mData.bindMemory(to: Int16.self, capacity: Int(inNumberFrames))
    let bufferPtr = UnsafeBufferPointer(start: ptrData, count: Int(inNumberFrames))

    count += 1
    addedBuffer += Array(bufferPtr)

    if count == 2 {

        let _ = TPCircularBufferProduceBytes(&circularBuffer, addedBuffer, UInt32(addedBuffer.count * 2))

        count = 0
        addedBuffer = []

        let buffer = TPCircularBufferTail(&circularBuffer, &availableBytes)

        memcpy(&targetBuffer, buffer, Int(min(bytesToCopy, Int(availableBytes))))

        TPCircularBufferConsume(&circularBuffer, UInt32(min(bytesToCopy, Int(availableBytes))))

        self.audioRecordingDelegate(inTimeStamp.pointee.mSampleTime / Double(16000), targetBuffer)


    }
}
return err;
 }

Here i am getting inNumberOfFrames almost 341 and i am appending 2 arrays together to get a bigger framesize (needed 640) for Android but i am only encoding 640 by the help of TPCircularBuffer.

func gotSomeAudio(timeStamp: Double, samples: [Int16]) {

samples.count))



    let encodedData = opusHelper?.encodeStream(of: samples)
OPUS_SET_BITRATE_REQUEST)


    let myData = encodedData!.withUnsafeBufferPointer {
        Data(buffer: $0)
    }

    var protoModel = ProtoModel()
    seqNumber += 1
    protoModel.sequenceNumber = seqNumber
    protoModel.timeStamp = Date().currentTimeInMillis()
    protoModel.payload = myData

    DispatchQueue.global().async {
        do {
            try self.tcpClient?.send(data: protoModel)
        } catch {
            print(error.localizedDescription)
        }
    }
    let diff = CFAbsoluteTimeGetCurrent() - start
                             print("Time diff is \(diff)")
}

In the above code i am opus encoding 640 frameSize and adding it to ProtoBuf payload and Sending it via UDP.

On Android side i am parsing the Protobuf and decoding the 640 framesize and playing it with AudioTrack.There is no problem with android side as i have recorded and played sound just by using Android but the problem comes when i record sound via IOS and play through Android Side.

Please don't suggest to increase the frameSize by setting Preferred IO Buffer Duration. I want to do it without changing this.

https://stackoverflow.com/a/57873492/12020007 It was helpful.

https://stackoverflow.com/a/58947295/12020007 I have updated my code according to your suggestion, removed the delegate and array concatenation but there is still clipping on android side. I have also calculated the time it takes to encode bytes that is approx 2-3 ms.

Updated callback code is

var err: OSStatus = noErr
        // we are calling AudioUnitRender on the input bus of AURemoteIO
        // this will store the audio data captured by the microphone in ioData
        err = AudioUnitRender(audioUnit!, ioActionFlags, inTimeStamp, 1, inNumberFrames, ioData)

        if let mData = ioData[0].mBuffers.mData {

            _ = TPCircularBufferProduceBytes(&circularBuffer, mData, inNumberFrames * 2)

            print("mDataByteSize: \(ioData[0].mBuffers.mDataByteSize)")
            count += 1

            if count == 2 {

                count = 0

                let buffer = TPCircularBufferTail(&circularBuffer, &availableBytes)

                memcpy(&targetBuffer, buffer, min(bytesToCopy, Int(availableBytes)))

                TPCircularBufferConsume(&circularBuffer, UInt32(min(bytesToCopy, Int(availableBytes))))

                let encodedData = opusHelper?.encodeStream(of: targetBuffer)


                let myData = encodedData!.withUnsafeBufferPointer {
                    Data(buffer: $0)
                }

                var protoModel = ProtoModel()
                seqNumber += 1
                protoModel.sequenceNumber = seqNumber
                protoModel.timeStamp = Date().currentTimeInMillis()
                protoModel.payload = myData

                    do {
                        try self.udpClient?.send(data: protoModel)
                    } catch {
                        print(error.localizedDescription)
                    }

            }

        }
        return err;

Answer 1

Your code is doing Swift memory allocation (Array concatenation) and Swift method calls (your recording delegate) inside the audio callback. Apple (in a WWDC session on Audio) recommends not doing any memory allocation or method calls inside the real-time audio callback context (especially when requesting short Preferred IO Buffer Durations). Stick to C function calls, such as memcpy and TPCircularBuffer.

Added: Also, don't discard samples. If you get 680 samples, but only need 640 for a packet, keep the 40 "left over" samples and use them appended in front of a later packet. The circular buffer will save them for you. Rinse and repeat. Send all the samples you get from the audio callback when you've accumulated enough for a packet, or yet another packet when you end up accumulating 1280 (2*640) or more.