3

For a given string, I'm trying to count the number of appearances of each word and emoji. I did it already here for emojis that consists only from 1 emoji. The problem is that a lot of the current emojis are composed from a few emojis.

Like the emoji ‍‍‍ consists of four emojis - ‍ ‍ ‍ , and emojis with human skin color, for example is etc.

The problem boils down to how to split the string in the right order, and then counting them is easy.

There are some good questions that addressed the same thing, like link1 and link2 , but none of them applies to the general solution (or the solution is outdated or I just can't figure it out).

For example, if the string would be hello ‍ emoji hello ‍‍‍, then I'll have {'hello':2, 'emoji':1, '‍‍‍':1, '‍':1} My strings are from Whatsapp, and all were encoded in utf8.

I had many bad attempts. Help would be appreciated.

sheldonzy
  • 5,505
  • 9
  • 48
  • 86

3 Answers3

3

Huge thanks Mark Tolonen. Now in order to count words and emojis and words in a given string, I'll use emoji.UNICOME_EMOJI in order to decide what is an emoji and what is not (from the emoji package), and then remove from the string the emojis.

Currently not an ideal answer, but it works and I'll edit if it will be changed.

import emoji
import regex
def split_count(text):
    total_emoji = []
    data = regex.findall(r'\X',text)
    flag = False
    for word in data:
        if any(char in emoji.UNICODE_EMOJI for char in word):  
            total_emoji += [word] # total_emoji is a list of all emojis

    # Remove from the given text the emojis
    for current in total_emoji:
        text = text.replace(current, '') 

    return Counter(text.split() + total_emoji)


text_string = "here hello world hello‍‍‍"    
final_counter = split_count(text_string)

Output:

final_counter
Counter({'hello': 2,
         'here': 1,
         'world': 1,
         '\u200d\u200d\u200d': 1,
         '': 5,
         '': 1})
sheldonzy
  • 5,505
  • 9
  • 48
  • 86
2

Use the 3rd party regex module, which supports recognizing grapheme clusters (sequences of Unicode codepoints rendered as a single character):

>>> import regex
>>> s='‍‍‍'
>>> regex.findall(r'\X',s)
['\u200d\u200d\u200d', '']
>>> for c in regex.findall('\X',s):
...     print(c)
... 
‍‍‍

To count them:

>>> data = regex.findall(r'\X',s)
>>> from collections import Counter
>>> Counter(data)
Counter({'\u200d\u200d\u200d': 1, '': 1})
Mark Tolonen
  • 166,664
  • 26
  • 169
  • 251
  • Thanks. What should I do when I include text in this string? Because when there are words in the string, it will count all the letters as well. – sheldonzy Mar 07 '18 at 16:01
  • @sheldonzy that's more difficult, because as you can see, emoji are complex and the are not made up of strictly code points from the emoji range of Unicode. – Mark Tolonen Mar 07 '18 at 16:05
  • Ok thanks. I added the complete function as an additional answer. Not sure this is the best work, but currently it works. – sheldonzy Mar 07 '18 at 18:05
0

emoji.UNICODE_EMOJI is a dictionary with structure

{'en': 
    {'': ':1st_place_medal:',
     '': ':2nd_place_medal:',
     '': ':3rd_place_medal:' 
... }
}

so you will need to use emoji.UNICODE_EMOJI['en'] for the above code to work.

William Egesdal
  • 1