1

I work on an iOS app that displays images that often contain text, and I'm adding support for ImageAnalysisInteraction as described in this WWDC 2022 session. I have gotten as far as making the interaction show up and being able to select text and get the system selection menu, and even add my own action to the menu via the buildMenuWithBuilder API. But what I really want to do with my custom action is get the selected text and do a custom lookup-like thing to check the text against other content in my app.

So how do I get the selected text from an ImageAnalysisInteraction on a UIImageView? The docs show methods to check if there is selected text, but I want to know what the text is.

Tom Hamming
  • 10,577
  • 11
  • 71
  • 145

2 Answers2

2

I was trying to solve the same problem. However, there doesn't currently seem to be any straightforward way to get selected text from ImageAnalysisInteraction. The closest thing seems to be the ImageAnalysis.transcript property, but it contains all the OCR text, not just what the user selected.

My solution was to capture the text whenever the user taps on the copy button on the selection menu. You can do this by observing clipboard changes, which allows you to copy the selected text from the clipboard whenever a change is detected.

See:

mrphuwin
  • 21
  • 4
0

Hope this help you

// Step -1
import Vision
// Step -2
// converting image into CGImage
guard let cgImage = imageWithText.image?.cgImage else {return}
// Step -3
// creating request with cgImage
let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
// Step -4
let request = VNRecognizeTextRequest { request, error in
     guard let observations = request.results as [VNRecognizedTextObservation],
     error == nil else {return}
     let text = observations.compactMap({
     $0.topCandidates(1).first?.string
     }).joined(separator: ", ")
     print(text) // text we get from image
}
   // step -5
    request.recognitionLevel = VNRequestTextRecognitionLevel
    try handler.perform([request])

For Reference and more details

El-Dow
  • 222
  • 2
  • 12
  • I found this. Article also https://betterprogramming.pub/enabling-live-text-interactions-with-images-in-swiftui-5dd1d7f1676 – El-Dow Jun 30 '22 at 22:11
  • What I really want is the text that the user has selected, though. I'd rather not have to re-do the whole image analysis. And if I did this, I'd have to get the coordinates of the selected text somehow and then check for text results at those coordinates. – Tom Hamming Jun 30 '22 at 22:18