How to apply Deeplab V3 in Xcode for real time segmentation?

Question

Actually i am a beginner in swift and Deeplab V3. I literally don't know how to integrate deeplab on Xcode. I only just want to use tensorflow trained example model for semantic segmentation in ios.

Answer 1

https://github.com/cainxx/image-segmenter-ios

Go through the above mentioned link which implements a COREML model used for Segmentation. Convert your Tensorflow model into Coreml model using standard conversion tools. We have tested it once. And so confident enough that it will work.

Answer 2

Apple offers deeplab v3 in .mlmodel format in its official site. U can integrate it in ur xcode easily by drag&drop. The only probleme i found is the output format is multiarray and i dont know how to show the result as an image. Here is the link for deeplab mlmodel https://docs-assets.developer.apple.com/coreml/models/Image/ImageSegmentation/DeepLabV3/DeepLabV3.mlmodel

Answer 3

To convert from the multiarray to an image you have to rasterize the data using cg. Here's sample that sets the alpha to 1 for every non zero value in the multi array. So, everything that is recognized in the multiarray has full alpha. You can interpret different values out of the array as well.

            guard let mlMultiArray = observations.first?.featureValue.multiArrayValue else {
            
                return
            }
            let aWidth = CGFloat(mlMultiArray.shape[0].intValue)
            let aHeight = CGFloat(mlMultiArray.shape[1].intValue)

            UIGraphicsBeginImageContext(
                CGSize(width: aWidth, height: aHeight))
            if let ctx = UIGraphicsGetCurrentContext() {

                ctx.clear(CGRect(x: 0.0, y: 0.0, width: Double(aWidth), height: Double(aHeight)));

                for j in 0..<Int(aHeight) {
                    for i in 0..<Int(aWidth) {

                        let aValue =
                            (mlMultiArray[j * Int(aHeight) + i].floatValue > 0.0) ?
                                1.0 : 0.0
                        let aRect = CGRect(
                            x: CGFloat(i),
                            y: CGFloat(j),
                            width: 1.0,
                            height: 1.0)

                        let aColor: UIColor = UIColor(
                            displayP3Red: 0.0,
                            green: 0.0,
                            blue: 0.0,
                            alpha: CGFloat(aValue))

                        aColor.setFill()
                        UIRectFill(aRect)
                    }
                }
                
                let anImage = UIGraphicsGetImageFromCurrentImageContext()

                self.segmentationImage = anImage
                UIGraphicsEndImageContext()
            }