How to convert Data of Int16 audio samples to array of float audio samples

Question

I'm currently working with audio samples. I get them from AVAssetReader and have a CMSampleBuffer with something like this:

guard let sampleBuffer = readerOutput.copyNextSampleBuffer() else {
guard reader.status == .completed else { return nil }
// Completed
// samples is an array of Int16
let samples = sampleData.withUnsafeBytes {
  Array(UnsafeBufferPointer<Int16>(
  start: $0, count: sampleData.count / MemoryLayout<Int16>.size))
 }

 // The only way I found to convert [Int16] -> [Float]...
 return samples.map { Float($0) / Float(Int16.max)}
}

guard let blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer) else {
return nil
}

let length = CMBlockBufferGetDataLength(blockBuffer)
let sampleBytes = UnsafeMutablePointer<UInt8>.allocate(capacity: length)
      CMBlockBufferCopyDataBytes(blockBuffer, 0, length, sampleBytes)

      sampleData.append(sampleBytes, count: length)
}

As you can see the only I found to convert [Int16] -> [Float] issamples.map { Float($0) / Float(Int16.max) but by doing this my processing time is increasing. Does it exist an other way to cast a pointer of Int16 to a pointer of Float?

If your intention is to convert the integers -32767 ... 32767 to -1.0 ... 1.0 then that's how it is done. But you don't need to create an Array, you can map the UnsafeBufferPointer. — Martin R, Aug 24 '17 at 13:49
How would you *"cast a pointer"*? 1) Int16 is 2 bytes and Float is 4 bytes. 2) You *compute* floating point values from the integers. — Martin R, Aug 24 '17 at 14:06
@MartinR In fact I would like the iteration to compute a floating point by doing `Int16value / Int16.max`. Isn't a way in Swift to say "Here is a pointer of Int16 could you give me the computed float pointer" ? When I used a UnsafeMutableRawPointer (a pointer of UInt8) I was able to do something like: `var floatValues = bytes.bindMemory(to: Float.self, capacity: bytesTotal)` — DEADBEEF, Aug 24 '17 at 14:15
Casting (or "rebinding" in Swift lingo) only changes the way how the memory is interpreted, it does not change the memory itself. — Martin R, Aug 24 '17 at 14:17

Martin R · Accepted Answer · 2017-08-24T15:27:10.300

"Casting" or "rebinding" a pointer only changes the way how memory is interpreted. You want to compute floating point values from integers, the new values have a different memory representation (and also a different size).

Therefore you somehow have to iterate over all input values and compute the new values. What you can do is to omit the Array creation:

let samples = sampleData.withUnsafeBytes {
    UnsafeBufferPointer<Int16>(start: $0, count: sampleData.count / MemoryLayout<Int16>.size)
}
return samples.map { Float($0) / Float(Int16.max) }

Another option would be to use the vDSP functions from the Accelerate framework:

import Accelerate
// ...

let numSamples = sampleData.count / MemoryLayout<Int16>.size
var factor = Float(Int16.max)
var floats: [Float] = Array(repeating: 0.0, count: numSamples)

// Int16 array to Float array:
sampleData.withUnsafeBytes {
    vDSP_vflt16($0, 1, &floats, 1, vDSP_Length(numSamples))
}
// Scaling:
vDSP_vsdiv(&floats, 1, &factor, &floats, 1, vDSP_Length(numSamples))

I don't know if that is faster, you'll have to check. (Update: It is faster, as ColGraff demonstrated in his answer.)

An explicit loop is also much faster than using map:

let factor = Float(Int16.max)
let samples = sampleData.withUnsafeBytes {
    UnsafeBufferPointer<Int16>(start: $0, count: sampleData.count / MemoryLayout<Int16>.size)
}
var floats: [Float] = Array(repeating: 0.0, count: samples.count)
for i in 0..<samples.count {
    floats[i] = Float(samples[i]) / factor
}
return floats

An additional option in your case might be to use CMBlockBufferGetDataPointer() instead of CMBlockBufferCopyDataBytes() into allocated memory.

Having some trouble with your first block of code, it's having trouble converting the `UnsafeBufferPointer.init` `start` parameter from `UnsafeRawBufferPointer` to `UnsafePointer<_>?` — , Aug 24 '17 at 15:12
@ColGraff: My `sampleData` has the type `Data`. I *guessed* that from `sampleData.withUnsafeBytes()` in the question. — Martin R, Aug 24 '17 at 15:18

score 2 · Answer 2 · 2017-08-24T15:18:06.900

2

You can do considerably better if you use the Accelerate Framework for the conversion:

import Accelerate

// Set up random [Int]
var randomInt = [Int16]()

randomInt.reserveCapacity(10000)
for _ in 0..<randomInt.capacity {
  let value = Int16(Int32(arc4random_uniform(UInt32(UInt16.max))) - Int32(UInt16.max / 2))
  randomInt.append(value)
}

// Time elapsed helper: https://stackoverflow.com/a/25022722/887210
func printTimeElapsedWhenRunningCode(title:String, operation:()->()) {
  let startTime = CFAbsoluteTimeGetCurrent()
  operation()
  let timeElapsed = CFAbsoluteTimeGetCurrent() - startTime
  print("Time elapsed for \(title): \(timeElapsed) s.")
}

// Testing

printTimeElapsedWhenRunningCode(title: "vDSP") {
  var randomFloat = [Float](repeating: 0, count: randomInt.capacity)
  vDSP_vflt16(randomInt, 1, &randomFloat, 1, vDSP_Length(randomInt.capacity))
}

printTimeElapsedWhenRunningCode(title: "map") {
  randomInt.map { Float($0) }
}

// Results
//
// Time elapsed for vDSP   : 0.000429034233093262 s.
// Time elapsed for flatMap: 0.00233501195907593 s.

It's an improvement of about 5 times faster.

(Edit: Added some changes suggested by Martin R)

edited Aug 24 '17 at 15:18

answered Aug 24 '17 at 14:51

Seems we had the same idea :) – Some remarks: Your code crashes because randomFloat is created *before* randomInt.capacity is set, that makes it an empty array. You can simply pass `&randomFloat` to vDSP_vflt16(). The vDSP measurement should include the randomFloat array creation. There is no reason to use flatMap, just `randomInt.map { Float($0) }` – Martin R Aug 24 '17 at 15:03
@MartinR Ahh yeah, I had moved the initialization of `randomFloat` to earlier to neaten up the code and neglected to test for that. I'll correct it and re-test. Thanks! – Aug 24 '17 at 15:08
Interestingly, `for i in 0.. – Martin R Aug 24 '17 at 15:20
@MartinR It's closer. I'm still seeing some improvement, maybe 50% to 75% time when using vDSP vs for loop. – Aug 24 '17 at 15:24
1

A bit more discussion of vDSP, caching, and the effects on testing in this chat: https://chat.stackoverflow.com/rooms/152775/swift-vdsp – Aug 24 '17 at 15:43

score 2 · Answer 3 · answered Aug 25 '17 at 16:34

@MartinR and @ColGraff gave really good answers, and thank you for everybody and the fast replies. however I found an easier way to do that without any computation. AVAssetReaderAudioMixOutput requires an audio settings dictionary. Inside we can set the key AVLinearPCMIsFloatKey: true. This way I will read my data like this

let samples = sampleData.withUnsafeBytes {
    UnsafeBufferPointer<Float>(start: $0, 
                               count: sampleData.count / MemoryLayout<Float>.size)
}

swift2geek · Answer 4 · 2017-08-24T14:03:18.613

0

for: Xcode 8.3.3 • Swift 3.1

extension Collection where Iterator.Element == Int16 {
    var floatArray: [Float] {
        return flatMap{ Float($0) }
    }
}

usage:

let int16Array: [Int16] = [1, 2, 3 ,4]    
let floatArray = int16Array.floatArray

edited Aug 24 '17 at 14:03

answered Aug 24 '17 at 13:59

swift2geek

1,697
1
20
27

Sorry maybe I ddin't explain well but I wanted to convert Int16 audio samples to float audio samples without re iterating the whole samples as you do in the floatArray method. – DEADBEEF Aug 24 '17 at 14:03

How to convert Data of Int16 audio samples to array of float audio samples

4 Answers4

Linked