-2

I can't find any example coding about how to do tokenization with csv file using Pandas Python. Below is my code with the cleaned review. I am going to use this "cleaned_review" to perform tokenization.

enter image description here

Jacky
  • 1
  • 1
  • 1

1 Answers1

0

to perform tokenization on cleaned_review, apply the NLTK word tokenizer row-wise to the column.

import nltk
steam_reviews['cleaned_review'] = steam_reviews.apply(lambda row: nltk.word_tokenize(row['cleaned_review']), axis=1)
thorntonc
  • 2,046
  • 1
  • 8
  • 20