Multiple workers in Swift Command Line Tool

Question

When writing a Command Line Tool (CLT) in Swift, I want to process a lot of data. I've determined that my code is CPU bound and performance could benefit from using multiple cores. Thus I want to parallelize parts of the code. Say I want to achieve the following pseudo-code:

Fetch items from database
Divide items in X chunks
Process chunks in parallel
Wait for chunks to finish
Do some other processing (single-thread)

Now I've been using GCD, and a naive approach would look like this:

let group = dispatch_group_create()
let queue = dispatch_queue_create("", DISPATCH_QUEUE_CONCURRENT)
for chunk in chunks {
    dispatch_group_async(group, queue) {
        worker(chunk)
    }
}
dispatch_group_wait(group, DISPATCH_TIME_FOREVER)

However GCD requires a run loop, so the code will hang as the group is never executed. The runloop can be started with dispatch_main(), but it never exits. It is also possible to run the NSRunLoop just a few seconds, however that doesn't feel like a solid solution. Regardless of GCD, how can this be achieved using Swift?

Answer 1

I mistakenly interpreted the locking thread for a hanging program. The work will execute just fine without a run loop. The code in the question will run fine, and blocking the main thread until the whole group has finished.

So say chunks contains 4 items of workload, the following code spins up 4 concurrent workers, and then waits for all of the workers to finish:

let group = DispatchGroup()
let queue = DispatchQueue(label: "", attributes: .concurrent)

for chunk in chunk {
    queue.async(group: group, execute: DispatchWorkItem() {
        do_work(chunk)
    })
}

_ = group.wait(timeout: .distantFuture)

Answer 2

Just like with an Objective-C CLI, you can make your own run loop using NSRunLoop.

Here's one possible implementation, modeled from this gist:

class MainProcess {
    var shouldExit = false

    func start () {
        // do your stuff here
        // set shouldExit to true when you're done
    }
}

println("Hello, World!")

var runLoop : NSRunLoop
var process : MainProcess

autoreleasepool {
    runLoop = NSRunLoop.currentRunLoop()
    process = MainProcess()

    process.start()

    while (!process.shouldExit && (runLoop.runMode(NSDefaultRunLoopMode, beforeDate: NSDate(timeIntervalSinceNow: 2)))) {
        // do nothing
    }
}

As Martin points out, you can use NSDate.distantFuture() as NSDate instead of NSDate(timeIntervalSinceNow: 2). (The cast is necessary because the distantFuture() method signature indicates it returns AnyObject.)

If you need to access CLI arguments see this answer. You can also return exit codes using exit().

Answer 3

Swift 3 minimal implementation of Aaron Brager solution, which simply combines autoreleasepool and RunLoop.current.run(...) until you break the loop:

var shouldExit = false
doSomethingAsync() { _ in
    defer {
        shouldExit = true
    }
}
autoreleasepool {
    var runLoop = RunLoop.current
    while (!shouldExit && (runLoop.run(mode: .defaultRunLoopMode, before: Date.distantFuture))) {}
}

Answer 4

I think CFRunLoop is much easier than NSRunLoop in this case

func main() {
    /**** YOUR CODE START **/
    let group = dispatch_group_create()
    let queue = dispatch_queue_create("", DISPATCH_QUEUE_CONCURRENT)
    for chunk in chunks {
        dispatch_group_async(group, queue) {
            worker(chunk)
        }
    }
    dispatch_group_wait(group, DISPATCH_TIME_FOREVER)
    /**** END **/
}


let runloop = CFRunLoopGetCurrent()
CFRunLoopPerformBlock(runloop, kCFRunLoopDefaultMode) { () -> Void in
    dispatch_async(dispatch_queue_create("main", nil)) {
        main()
        CFRunLoopStop(runloop)
    }
}
CFRunLoopRun()