Python fastest way to remove single spaces from spaced out letters in string

Question

I have a document with some lines that have spaced out letters which I want to remove.

The problem is, that the strings are not following all the same rules. So I have some with just one space, also between the words and some with two or three speaces between the words

Examples:

"H e l l o g u y s"
"H e l l o  g u y s"
"H e l l o   g u y s"

all the above should be converted to --> "Hello guys"

"T h i s i s P a g e 1"  -->  "This is Page 1"

I wrote a script to remove every second space but not if next letter is numeric or capital. It's working almost OK, since the processed text is German and almost every time the words begin with capital letters... almost. Anyways I'm not satisfied with it. So I'm asking if there is a neat function for my problem.

text = text.strip()                     # remove spaces from start and end
out = text
if text.count(' ') >= (len(text)/2)-1:
    out = ''
    idx = 0
    for c in text:
        if c != ' ' or re.match('[0-9]|\s|[A-Z0-9ÄÜÖ§€]', text[idx+1]) or (idx > 0 and text[idx-1] == '-'):
            out += c
        idx += 1
text = out

Answer 1

You can check whether a word is a english word and then split the words. You could use a dedicated spellchecking library like PyEnchant.

For example:

import enchant
d = enchant.Dict("en_US")
d.check("Hello")

This will be a good starter. But there is the problem with "Expertsexchange".

Answer 2

Converting "H e l l o g u y s" might be very hard or not under the scope of this site. but if you wont to convert the strings like "H e l l o g u y s" or other that the number of spaces between words is different from spaces between letters you can use a the following code :

>>> import re
>>> s1="H e l l o  g u y s"
>>> s2="H e l l o   g u y s"
>>> ' '.join([''.join(i.split()) for i in re.split(r' {2,}',s2)])
'Hello guys'
>>> ' '.join([''.join(i.split()) for i in re.split(r' {2,}',s1)])
'Hello guys'

this code use a regular expression (' {2,}') for split the words . that split the string from where that have more than 2 spaces !

Answer 3

Not the most original answer but I've seen that your problem almost matches this one. I have taken unutbu's answer, slightly modified it to solve your queries with enchant. If you have any other dictionary, you can use that instead.

import enchant
d = enchant.Dict("en_US") # or de_DE

def find_words(instring, prefix = ''):
    if not instring:
        return []

    if (not prefix) and (d.check(instring)):
        return [instring]
    prefix, suffix = prefix + instring[0], instring[1:]
    solutions = []
    # Case 1: prefix in solution
    if d.check(prefix):
        try:
            solutions.append([prefix] + find_words(suffix, ''))
        except ValueError:
            pass
    # Case 2: prefix not in solution
    try:
        solutions.append(find_words(suffix, prefix))
    except ValueError:
        pass
    if solutions:            
        return sorted(solutions,
                      key = lambda solution: [len(word) for word in solution],
                      reverse = True)[0]

    else:
        raise ValueError('no solution')

inp = "H e l l o   g u y s T h i s i s P a g e 1" 
newInp = inp.replace(" ", "")

print(find_words(newInp))

This outputs:

['Hello', 'guys', 'This', 'is', 'Page', '1']

The linked page certainly is a good starting point for some pragmatic solutions. However, I think a proper solution should use n-grams. This solution could be modified to make use of multiple whitespaces as well, since they might indicate the presence of a word boundary.

Edit: You can also have a look at Generic Human's solution using a dictionary with relative word frequencies.

Answer 4

Demo

This is an algorithm that could do it. Not battle-tested, but just an idea.

d = ['this', 'is', 'page', 'hello', 'guys']
m = ["H e l l o g u y s", "T h i s i s P a g e 1", "H e l l o   g u y s", "H e l l o  g u y s"]
j = ''.join(m[0].split()).lower()

temp = []
fix = []


for i in j:
    temp.append(i)
    s = ''.join(temp) 

    if s in d:
        fix.append(s)       
        del temp[:]
    
    if i.isdigit():
        fix.append(i)
    
print(' '.join(fix))

Prints the following:

this is page 1, hello guys with your supplied test inputs.

Extending

You can use this dictionary which has words on each line, convert it to a list and play around from there.

Issues

As Martjin suggested, what would you do when you encounter "E x p e r t s e x c h a n g e". Well, in such scenarios, using n-gram probabilities would be an appropriate solution. For this you would have to look into NLP (Natural Language Processing) but I assume you don't want to go that far.

Answer 5

You cannot do this - the situation where valid word boundaries are represented the same way as spaces which should be removed is theoretically the same situation where you have no spaces at all in the text.

So you can "reduce" your problem to the problem of re-inserting word boundary spaces in a text with no spaces at all - which is just as impossible, because even with a dictionary containing every valid word - which you do not have -, you can either go for a greedy match and insert too few spaces, or go for a non-greedy match and insert too many.