Sort Cyrillic strings before Latin in python

Question

In my database, I have records in both Cyrillic and Latin characters. By default, they are listed alphabetically with Latin records first:

abc... bcd... cde... абв...

I would like to put the Cyrillic to the first place:

абв... abc... bcd... cde...

What I have tried so far:

1. This solution. It is not so great because it only sorts by the first word, and I can have both Cyrillic and Latin words in the same string (or even mixed characters in the same word).

2. Writing my own lists with Cyrillic and Latin alphabets. It works but is not great at all. I cannot take into account all possible letters in the two alphabets, including those with diacritics and write them down.

I have also been looking into PyICU but don't see how I can put it to use.

My guess is that I should use some custom collation here. The question is how this can be done in practice.

Answer 1

One way to do this is to use the transliterate module or maybe cytranslit and use a sort key that transliterates everything to the desired alphabet:

import transliterate

items = ['abc', 'bcd', 'cde', 'абв']

print(sorted(items, key=lambda x: transliterate.translit(x, 'ru')))

The output is the desired

['абв', 'abc', 'bcd', 'cde']

Answer 2

IMO this is not a trivial thing. I'd say that a collation is indeed required.

So, say a key function would convert a string to a tuple of codepoints, where all non-Cyrillic code points would be shifted by 100000):

import unicodedata

def key(s):
    SHIFT = 100000
    return tuple(
        ord(c) if is_cyrillic(c) else ord(c) + SHIFT
        for c in s
    )

def is_cyrillic(c):
    return unicodedata.name(c).startswith('CYRILLIC')        


>>> sorted(('wannt', 'waюnnt'), key=key)
Out[34]: ['waюnnt', 'wannt']

is_cyrillic can be optimized by using a preliminary table or caching the Cyrillic characters from the database strings.

Answer 3

You can try to generate a key by prepending every character with 1 if it's a Latin character and with 0 otherwise:

sorted(items, key = lambda item : ['1' + x if x < '\x7f' else '0' + x for x in item])

Answer 4

Very dirty but working solution for sorting according the first character. It also eliminates difference in register of letters.

ls = ['32', '24', 'xyz', 'WYZ', 'abc', 'абв', 'КЛМ', 'эюя', 'еёж', 'ёжз', '', '_']

def sort_rule(st):
    st = st.lower()
    ch = st[0] if st else ''       
    if ch >= 'а' and ch <= 'я':
        st = '1' + st
    elif ch == 'ё':
        st = '1е' + st
    elif ch >= 'a' and ch <= 'z':
        st = '2' + st
    else:
        st = '3' + st
    return st

sorted(ls, key=sort_rule)

> ['абв', 'еёж', 'ёжз', 'КЛМ', 'эюя', 'abc', 'WYZ', 'xyz', '', '24', '32', '_']

For comparison, default sorting gives the next result:

sorted(ls)

> ['', '24', '32', 'WYZ', '_', 'abc', 'xyz', 'КЛМ', 'абв', 'еёж', 'эюя', 'ёжз']

Answer 5

This in an old question, with four answers, but I feel the following answer is more appropriate.

If you have both icu4c and PyICU installed, there is a fairly simple solution. ICU uses the CLDR Collation Algorithm, a tailoring of the Unicode Collation Algorithm.

Since you want a language insensitive sort with all Cyrillic characters sorted first, the simplest approach is to tailor the CLDR root collation. All tailorings of ICU collations are tailorings of the root collation, and you provide the minimum changes required.

In this scenario, you just need a reorder directive:

from icu import RuleBasedCollator
items = ['cde', 'abc', 'bcd', 'абв']
rules = "[reorder Cyrl]"
collator = RuleBasedCollator(rules)
sorted(items, key=collator.getSortKey)
# ['абв', 'abc', 'bcd', 'cde']

But the above rules do exactly what the Russian collation rules do, reordering Cyrillic based on the root collation.

from icu import Locale, Collator
items = ['cde', 'abc', 'bcd', 'абв']
collator = Collator.createInstance(Locale("ru"))
print(sorted(items, key=collator.getSortKey))