Remove all spaces for chinese characters while keeping necessary spaces for english in Python regex

Question

Let's say my dataframe has column which is mixed with english and chinese words or characters, I would like to remove all the whitespaces between them if they're chinese words, otherwise if they're english, then keep one space only between words:

I have found a solution for removing extra spaces between english from here

import re
import pandas as pd

s = pd.Series(['V e  r y calm', 'Keen and a n a l y t i c a l',
'R a s h and careless', 'Always joyful', '你 好', '黑 石  公 司', 'FAN     STUD1O', 'beauty face 店  铺'])

Code:

regex = re.compile('(?<![a-zA-Z]{2})(?<=[a-zA-Z]{1}) +(?=[a-zA-Z] |.$)')
s.str.replace(regex, '')

Out:

Out[87]: 
0              Very calm
1    Keen and analytical
2      Rash and careless
3          Always joyful
4                    你 好
5               黑 石  公 司
dtype: object

But as you see, it works out for english but didn't remove spaces between chinese, how could get an expected result as follows:

Out[87]: 
0              Very calm
1    Keen and analytical
2      Rash and careless
3          Always joyful
4                    你好
5                 黑石公司
dtype: object

Reference: Remove all spaces between Chinese words with regex

Answer 1

You could use the Chinese (well, CJK) Unicode property \p{script=Han} or \p{Han}.
However, this only works if the regex engine supports UTS#18 Unicode regular expressions. The default Python re module does not but you can use the alternative (much improved) regex engine:

import regex as re

rex = r"(?<![a-zA-Z]{2})(?<=[a-zA-Z]{1})[ ]+(?=[a-zA-Z] |.$)|(?<=\p{Han}) +"
test_str = ("V e  r y calm\n"
    "Keen and a n a l y t i c a l\n"
    "R a s h and careless\n"
    "Always joyful\n"
    "你 好\n"
    "黑 石  公 司")
result = re.sub(rex, "", test_str, 0, re.MULTILINE | re.UNICODE)

Results in

Very calm
Keen and analytical
Rash and careless
Always joyful
你好
黑石公司

Online Demo (the demo is using PCRE for demonstration purposes only)

Answer 2

Use word boundaries \b in look arounds:

(?<=\b\w\b) +(?=\b\w\b)

This matches spaces between solitary (bounded by word boundaries) "word characters", which includes Chinese characters.

Pre python 3 (and for java for example), \w only matches English letters, so you would need to add the unicode flag (?u) to the front of the regex.

---

s = ['V e  r y calm', 'Keen and a n a l y t i c a l',
'R a s h and careless', 'Always joyful', '你 好', '黑 石  公 司']
regex = r'(?<=\b\w\b) +(?=\b\w\b)'
res = [re.sub(regex, '', line) for line in s]
print(res)

Output:

['Very calm', 'Keen and analytical', 'Rash and careless', 'Always joyful', '你好', '黑石公司']

Answer 3

This regex should get you what you want. See the full code snippet at the bottom.

regex = re.compile(
    "((?<![a-zA-Z]{2})(?<=[a-zA-Z]{1})\s+(?=[a-zA-Z]\s|.$)|(?<=[\u4e00-\u9fff]{1})\s+)",
    re.UNICODE,
)

I made the following edits to your regex above: Right now, the regex basically matches all spaces that appear after a single-letter word and before another single character word.

1. I added a part at the end of the regex that would select all spaces after a Chinese character (I used the unicode range [\u4e00-\u9fff] which would cover Japanese and Korean as well.
2. I changed the spaces in the regex to the whitespace character class \s so we could catch other input like tabs.
3. I also added the re.UNICODE flag so that \s would cover unicode spaces as well.

import re
import pandas as pd

s = pd.Series(
    [
        "V e  r y calm",
        "Keen and a n a l y t i c a l",
        "R a s h and careless",
        "Always joyful",
        "你 好",
        "黑 石  公 司",
    ]
)

regex = re.compile(
    "((?<![a-zA-Z]{2})(?<=[a-zA-Z]{1})\s+(?=[a-zA-Z]\s|.$)|(?<=[\u4e00-\u9fff]{1})\s+)",
    re.UNICODE,
)
s.str.replace(regex, "")

Output:

0              Very calm
1    Keen and analytical
2      Rash and careless
3          Always joyful
4                     你好
5                   黑石公司
dtype: object