-1

I'd like to extract a conversation for a game with JS (plain) and get just the text bubbles. This isn't that hard just to crop and image, but sadly the image's dimensions and ratios is going to change. Here's the image before and after

Original (messages removed):

screen shot

And then the auto cropped:

auto cropped

How do I crop this? All the text is preserved but nothing else is in it but the text (I need this because I'm converting it to text data via tesseract and it gets confused on some other things in the image). My goal is to have this entirely client side so no PHP could be used. The image ratio and dimensions are also going to change. It's always going to be landscape but some are practically square while others are really long and short. Is this possible. Could I use some sort of AI model to capture it? Any help is greatly appreciated. Thanks!

Edit

This game is Brawl Stars avaivible on iOS and the Play Store. Tapping system would be very hard, it’s made by Supercell which in my experience has their apps fairly well locked down. All apis are foucues on battle results not the conversations.

Community
  • 1
  • 1
Jack
  • 1,893
  • 1
  • 11
  • 28
  • It would really help if you'd add some code to your question... – Dragos Rizescu Nov 05 '18 at 23:39
  • @DragosRizescu It's hard too. I don't know which library would do this. – Jack Nov 05 '18 at 23:42
  • if you're capturing the chat why not capture the exact text from the bubbles to begin with? – Paul G Mihai Nov 05 '18 at 23:43
  • @PaulGMihai I'm not sure I'm following you. If I just use the pure screenshot (the only input) to tesseract it makes a lot of errors and the conversation is practically irreplaceable from the other text. If I were to use the cropped image I does the conversion perfectly. – Jack Nov 05 '18 at 23:47
  • 2
    Wrong approach, your screen scraping by literally taking a picture of the screen, attempting to crop it and then running it through ocr software, instead of intercepting or hooking into the api response from the games server which you would have your text as-is, no AI needed. – Lawrence Cherone Nov 05 '18 at 23:49
  • yes but why go though all the trouble? Why not hook into the game chat api or capture from the browser the exact text via js? – Paul G Mihai Nov 05 '18 at 23:56
  • @PaulGMihai I did some checking and the only apis that I could find didn't relate to this, they focused on different areas. I could add a bot account but it wouldn't work because of the player limit pre room. – Jack Nov 06 '18 at 00:00
  • The text also cannot be copied and pasted if that's what you were referring too. – Jack Nov 06 '18 at 00:01
  • It is really difficult for anyone to give meaningful advice without knowing more about the situation. Is this an HTML5 game? Unity? Can you navigate to it from a desktop browser? Or is it an Android or ios game? What is the game called? (is there a url?) This OCR method should probably be your last resort. Even if you don't have access to the game's JS API (assuming it's HTML5), you might still be able to listen in on a port and get the text that way. – Ringo Nov 06 '18 at 00:15
  • @Ringo thanks, could you check my edit? – Jack Nov 06 '18 at 02:00

1 Answers1

0

Before I answer I should note a few things. Firstly, this is a pretty terrible way to do things, hopefully there is some javascript you can hook into, or maybe it even displays things to the DOM. I don't know the game, so I can't tell you, but you should definately look for alternate solutions than just scraping the visuals. Secondly: without knowing what libraries or languages or code that you're using, it's a bit tricky to provide example code, but...

The best way to do this would probably be first looking for easily identifiable corners of the screen, and using those to get the area that you're looking for. Something like that top left corner where it meets the top bar. Maybe get the bottom right from the speech bubble or just use the literal bottom right hand corner.

Alternatively you could try to figure out how it scales and edit your code appropriately, but that isn't a fun approach...

Sam Clarke
  • 120
  • 1
  • 11
  • Thanks, and sorry for the vaugness, would there be some sort of libary that would do this? – Jack Nov 05 '18 at 23:56
  • @JackStoller you should like wait until you get more responses before marking this one correct. – Ringo Nov 06 '18 at 00:04
  • See https://stackoverflow.com/questions/3351122/what-is-the-best-javascript-image-processing-library There's also https://trackingjs.com/ which isn't mentioned there. I don't typically do this in javascript, so I'm not entirely sure which ones the "best". – Sam Clarke Nov 06 '18 at 02:43