Adding to array in parallel

Question

I'm using Grand Central Dispatch to transforms elements of one array into another. I call dispatch_apply on the source array, transform it into zero or more items, then add them to the destination array. This is a simplified example:

let src = Array(0..<1000)
var dst = [UInt32]()

let queue = dispatch_queue_create("myqueue", DISPATCH_QUEUE_CONCURRENT)
dispatch_apply(src.count, queue) { i in
    dst.append(arc4random_uniform(UInt32(i))) // <-- potential error here
}

print(dst)

I sometimes get an error on the append line. The error is always one of:

1. malloc: *** error for object 0x107508f00: pointer being freed was not allocated
2. fatal error: UnsafeMutablePointer.destroy with negative count
3. fatal error: Can't form Range with end < start

I guess this is due to append not being thread-safe. What did I do wrong and how to fix it?

Compare http://stackoverflow.com/questions/26693838/process-array-in-parallel-using-gcd. — Martin R, Nov 04 '15 at 02:34
Thanks @MartinR. That question alter the existing value of the array rather than append to it. `withUnsafeMutablePointer` does not have an `append` method — Jenny, Nov 04 '15 at 02:56
As you already noticed, append() is not thread-safe. You probably have to create the array with the necessary size first. Then you can use the methods from the referenced Q&A to fill the array in parallel from multiple threads. — Martin R, Nov 04 '15 at 06:08
sometimes? man, you are lucky!. anyway, as you wrote, swift types are not thread-safe. i thing, that this is just very simplified example of something what you are trying to do. try to avoid this approach. even though you will have a thread-safe version of an array, this approach will be real performance killer. if you need just separate a long running 'transformation' from your main loop, call the whole 'transformation' asynchronously, but as one or few separate blocks — user3441734, Nov 04 '15 at 06:45

score 3 · Accepted Answer · answered Nov 10 '15 at 11:58

As you have discovered, Swift may relocate the array for memory efficiency. When you run multiple append, it's bound to happen. My guess is that appending to the array is a very low cost operation compared to your transformation. You can create a serial queue just to extend the dst array:

let src = Array(0..<1000)
var dst = [UInt32]()

let queue = dispatch_queue_create("myqueue", DISPATCH_QUEUE_CONCURRENT)
let serialQueue = dispatch_queue_create("mySerialQueue", DISPATCH_QUEUE_SERIAL)
let serialGroup = dispatch_group_create()

dispatch_apply(src.count, queue) { i in
    let result = arc4random_uniform(UInt32(i)) // Your long-running transformation here
    dispatch_group_async(serialGroup, serialQueue) {
        dst.append(result)
    }
}

// Wait until all append operations are complete
dispatch_group_wait(serialGroup, DISPATCH_TIME_FOREVER)

Why did you use `dispatch_group_async` instead of `group_async`? — Jenny, Nov 10 '15 at 13:35
I think you meant `dispatch_group_async` vs. `dispatch_async`. The reason is that you can wait on the first, but not the second function. — Code Different, Nov 11 '15 at 01:49
I have this same issue, but i didn't think it would be something so complex XD thanks for the solution ! — A'sa Dickens, Aug 25 '16 at 13:51

Lukas · Answer 2 · 2018-11-16T09:41:12.340

In case anyone is interested in Swift 4 (I suppose Swift 3 onwards) solution (slightly different approach):

let source = Array(0..<1000)

// If inside class/struct, make this lazy or execute the block inside init
var destination: [UInt32] = {
    var copy = Array<UInt32>.init(repeating: 0, count: source.count)

    DispatchQueue.concurrentPerform(iterations: source.count) { i in
        // Do stuff to get the i'th element 
        copy[i] = UInt32.random(in: 0..<i) 
    }

    return copy
}()

If you choose to avoid the tie between your array size and the number of concurrent operations:

(e.g. you may choose to do it in two iterations 0..<500 in parallel with 500..<1000)

let iterations = 2  

DispatchQueue.concurrentPerform(iterations: iterations) { index in

    // First iteration would start at 0 whereas second (index = 1) starts at 500
    let start = index * source.count / iterations

    // First and second iterations will have 500 and 1000 upper bounds respectively 
    let end = (index + 1) * source.count / iterations

    for i in start..<end {
        // Do stuff to get the i'th element
        copy[i] = UInt32.random(in: 0..<source.count)
    }
}

Adding to array in parallel

2 Answers2

Linked