1

I have a dataframe which contain phrases and I want to extract only compound words separated by a hyphen from the dataframe and place them in another dataframe.

df=pd.DataFrame({'Phrases': ['Trail 1 Yellow-Green','Kim Jong-il was here', 'President Barack Obama', 'methyl-butane', 'Derp da-derp derp', 'Pok-e-mon'],})

So far here is what I got so far:

import pandas as pd

df=pd.DataFrame({'Phrases': ['Trail 1 Yellow-Green','Kim Jong-il was here', 'President Barack Obama', 'methyl-butane', 'Derp da-derp derp', 'Pok-e-mon'],})


new = df['Phrases'].str.extract("(?P<part1>.*?)-(?P<part2>.*)")

results

>>> new
            part1        part2
0  Trail 1 Yellow        Green
1        Kim Jong  il was here
2             NaN          NaN
3          methyl       butane
4         Derp da    derp derp
5             Pok        e-mon

What I want is to have just the word so it would be(note that Pok-e-mon appears as Nan due to 2 hyphens):

>>> new
            part1        part2
0          Yellow        Green
1             Jong          il
2             NaN          NaN
3          methyl       butane
4              da         derp
5             NaN          NaN
ccsv
  • 8,188
  • 12
  • 53
  • 97

2 Answers2

1

You can use this regex:

(?:[^-\w]|^)(?P<part1>[a-zA-Z]+)-(?P<part2>[a-zA-Z]+)(?:[^-\w]|$)

(?:               # non capturing group
    [^-\w]|^        # a non-hyphen or the beginning of the string
)
(?P<part1>
    [a-zA-Z]+     # at least a letter
)-(?P<part2>
    [a-zA-Z]+
)
(?:[^-\w]|$)        # either a non-hyphen character or the end of the string
  • Your first problem is that nothing prevents the . from eating up spaces. [a-zA-Z] only select letters so it will avoid "jumping" from one word to another.
  • For the pok-e-mon case, you need to check that there isn't a hyphen right before or after your match.

See Demo here

Robin
  • 9,415
  • 3
  • 34
  • 45
  • I tried this and something similar it does not work with the pandas `str.extract` if I use this it will return all Nans – ccsv Apr 17 '14 at 16:42
  • @ccsv: Does the central `(?P[a-zA-Z]+)-(?P[a-zA-Z]+)` part works (except on the pokemon case)? I edited to replace the word boundaries, maybe pandas doesn't like those. – Robin Apr 17 '14 at 16:50
  • Ok I guess there is no other short way to capture it all in one go so I will make another regex for the pokemon and similar case so I can toss them out. I also came to the same regex code as you currently have. – ccsv Apr 17 '14 at 22:31
1

Given the specs I don't see where your first line of Nan, Nan is coming from. Maybe it is typo in your example? Anyways, here is a possible solution.

import re

# returns words with at least one hyphen
def split_phrase(phrase):
    return re.findall('(\w+(?:-\w+)+)', phrase)

# get all words with hyphens
words_with_hyphens = sum(df.Phrases.apply(split_phrase).values)
# split all words into parts
split_words = [word.split('-') for word in words_with_hyphens]
# keep words with two parts only, else return (Nan, Nan)
new_data = [(ws[0], ws[1]) if len(ws) == 2 else (np.nan, np.nan) for ws in split_words]
# create the new DataFrame
pd.DataFrame(new_data, columns=['part1', 'part2'])

#  part1   | part2
#------------------
# 0 Yellow | Green
# 1 Jong   | il
# 2 methyl | butane
# 3 da     | derp
# 4 NaN    | NaN
Matt
  • 17,290
  • 7
  • 57
  • 71