Unable to tokenize multiple columns in a dataframe

Question

I have a table with both numeric and string data but in separate columns. The table is answers to a web form and contains empty cells. I want to use text processing on the string columns. I cannot drop the rows with empty cells so for the empty string columns, I replaced the NaN with aplhabet 'a'.

Sample data

colmun_name1    column_name2     column_name3 column_name4 classify
This is a cat   This is a dog    1            2            0
This is a rat   This is a mouse  45           32           1
a               Good mouse       0            0            0

I used the following code to make sure all data in the string columns is actually string data.

df2=df[[column_name1, column_name2]]
for i in range(0,len(df2)):
cell=df2.iloc[i]
cell=str(str)
df2.iloc[i]=cell

Then when I tokenize, I get an error

    <ipython-input-64-24a99733ba19> in <module>
      1 from nltk.tokenize import word_tokenize
----> 2 tokenized_word=word_tokenize(df2)
      3 print(tokenized_word)

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/__init__.py in word_tokenize(text, language, preserve_line)
    126     :type preserver_line: bool
    127     """
--> 128     sentences = [text] if preserve_line else sent_tokenize(text, language)
    129     return [token for sent in sentences
    130             for token in _treebank_word_tokenizer.tokenize(sent)]

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/__init__.py in sent_tokenize(text, language)
     93     """
     94     tokenizer = load('tokenizers/punkt/{0}.pickle'.format(language))
---> 95     return tokenizer.tokenize(text)
     96 
     97 # Standard word tokenizer.

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/punkt.py in tokenize(self, text, realign_boundaries)
   1239         Given a text, returns a list of the sentences in that text.
   1240         """
-> 1241         return list(self.sentences_from_text(text, realign_boundaries))
   1242 
   1243     def debug_decisions(self, text):

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/punkt.py in sentences_from_text(self, text, realign_boundaries)
   1289         follows the period.
   1290         """
-> 1291         return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
   1292 
   1293     def _slices_from_text(self, text):

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/punkt.py in <listcomp>(.0)
   1289         follows the period.
   1290         """
-> 1291         return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
   1292 
   1293     def _slices_from_text(self, text):

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/punkt.py in span_tokenize(self, text, realign_boundaries)
   1279         if realign_boundaries:
   1280             slices = self._realign_boundaries(text, slices)
-> 1281         for sl in slices:
   1282             yield (sl.start, sl.stop)
   1283 

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/punkt.py in _realign_boundaries(self, text, slices)
   1320         """
   1321         realign = 0
-> 1322         for sl1, sl2 in _pair_iter(slices):
   1323             sl1 = slice(sl1.start + realign, sl1.stop)
   1324             if not sl2:

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/punkt.py in _pair_iter(it)
    311     """
    312     it = iter(it)
--> 313     prev = next(it)
    314     for el in it:
    315         yield (prev, el)

/anaconda3/lib/python3.6/site-packages/nltk/tokenize/punkt.py in _slices_from_text(self, text)
   1293     def _slices_from_text(self, text):
   1294         last_break = 0
-> 1295         for match in self._lang_vars.period_context_re().finditer(text):
   1296             context = match.group() + match.group('after_tok')
   1297             if self.text_contains_sentbreak(context):

TypeError: expected string or bytes-like object

I tried changing

df2=df[column_name1][column_name2]

But I get the same error.

What should I do?

score 1 · Accepted Answer · answered Dec 17 '18 at 14:08

Please see How to apply NLTK word_tokenize library on a Pandas dataframe for Twitter data?

TL;DR

# Creates a `colmun_name1_tokenized` column by 
# taking the `colmun_name1` column and 
# applying the word_tokenize function on every cell in the column. 

>>> df['colmun_name1_tokenized'] = df['colmun_name1'].apply(word_tokenize)

>>> df.head()
    colmun_name1     column_name2  column_name3  column_name4  classify  \
0  This is a cat    This is a dog             1             2         0   
1  This is a rat  This is a mouse            45            32         1   
2              a       Good mouse             0             0         0   

  colmun_name1_tokenized  
0     [This, is, a, cat]  
1     [This, is, a, rat]  
2                    [a]

If you need more than one column to be tokenized and you want to overwrite the column with the tokenized output:

>>> with StringIO(file_str) as fin:
...     df = pd.read_csv(fin, sep='\t')
... 
>>> for col_name in ['colmun_name1', 'column_name2']:
...     df[col_name] = df[col_name].apply(word_tokenize)
... 
>>> df.head()
         colmun_name1          column_name2  column_name3  column_name4  \
0  [This, is, a, cat]    [This, is, a, dog]             1             2   
1  [This, is, a, rat]  [This, is, a, mouse]            45            32   
2                 [a]         [Good, mouse]             0             0   

   classify  
0         0  
1         1  
2         0

Just the code:

from io import StringIO

import pandas as pd

from nltk import word_tokenize

file_str = """colmun_name1\tcolumn_name2\tcolumn_name3\tcolumn_name4\tclassify
This is a cat\tThis is a dog\t1\t2\t0
This is a rat\tThis is a mouse\t45\t32\t1
a\tGood mouse\t0\t0\t0 """

with StringIO(file_str) as fin:
    df = pd.read_csv(fin, sep='\t')

for col_name in ['colmun_name1', 'column_name2']:
    df[col_name] = df[col_name].apply(word_tokenize)

James Fulton · Answer 2 · 2018-12-17T13:56:31.630

0

I think your mistake is pretty simple replace cell=str(str) with cell=str(cell).

Also you need to have correct indentation and you can't call str on a row, only on individual cells. So your code should look something like this minimal example

import pandas as pd

data_dict = {'a':[l for l in 'aakjnasnkdf']+[None], 
               'b':[l for l in 'aakjnasnkdf']+[1], 
               'c':range(12)}

df=pd.DataFrame(data_dict)

column_name1 ='a'
column_name2 =  'b'
df2=df.loc[:,[column_name1, column_name2]]

for i in range(0,len(df2)):
    cell1, cell2 = df2.iloc[i]
    cell1=str(cell1)
    cell2 = str(cell2)
    df2.iloc[i]=[cell1,cell2]

edited Dec 17 '18 at 13:56

answered Dec 17 '18 at 09:31

James Fulton

322
2
8

Thanks @James Fulton. Unfortunately I end up with the same error – ratmatazz1123 Dec 17 '18 at 09:48
I realise that in my example of the correct indented code I didn't change `cell=str(str)` to `cell=str(cell)`. Will edit my answer above to add this. I also realise that when I tested it I used only one column. You can't apply a change to two columns in this way. Try doing them one at a time. Will update code above to include this too. – James Fulton Dec 17 '18 at 13:41

Unable to tokenize multiple columns in a dataframe

2 Answers2

TL;DR