How to read and label line by line a text file using nltk.corpus in Python

Question

My problem is to classify documents given two training data good_reviews.txt and bad_reviews.txt. So to start I need to load and label my training data where every line is a document itself which corresponds to a review. So my main task is to classify reviews (lines) from a given testing data.

I found a way how to load and label names data as follow:

from nltk.corpus import names
names = ([(name, 'male') for name in names.words('male.txt')] +
         [(name, 'female') for name in names.words('female.txt')])

So what I want to have is a similar thing which labels lines and not words. I am expecting that the code would be something like this which of course doesn't work since .lines is an invalid syntax:

reviews = ([(review, 'good_review') for review in reviews.lines('good_reviews.txt')] +
           [(review, 'bad_review') for review in reviews.lines('bad_reviews.txt')])

and I would like to have a result like this:

>>> reviews[0]
('This shampoo is very good blablabla...', 'good_review')

So, did you try that? What was the result? Where is your code, and exactly what is the problem with it? — jonrsharpe, Apr 27 '14 at 21:18
No it did not work because `.lines` is an invalid syntax and not defined in nltk.corpus. — user2987, Apr 27 '14 at 21:30

score 2 · Accepted Answer · edited May 23 '17 at 12:08

If you're reading your own textfile, then there's nothing much to do with NLTK, you can simply use file.readlines():

good_reviews = """This is great!
Wow, it amazes me...
An hour of show, a lifetime of enlightment
"""
bad_reviews = """Comme si, Comme sa.
I just wasted my foo bar on this.
An hour of s**t, ****.
"""
with open('/tmp/good_reviews.txt', 'w') as fout:
    fout.write(good_reviews)
with open('/tmp/bad_reviews.txt', 'w') as fout:
    fout.write(bad_reviews)

reviews = []
with open('/tmp/good_reviews.txt', 'r') as fingood, open('/tmp/bad_reviews.txt', 'r') as finbad:
    reviews = ([(review, 'good_review') for review in fingood.readlines()] + [(review, 'bad_review') for review in finbad.readlines()])

print reviews

[out]:

[('This is great!\n', 'good_review'), ('Wow, it amazes me...\n', 'good_review'), ('An hour of show, a lifetime of enlightment\n', 'good_review'), ('Comme si, Comme sa.\n', 'bad_review'), ('I just wasted my foo bar on this.\n', 'bad_review'), ('An hour of s**t, ****.\n', 'bad_review')]

If you're going to use the NLTK movie review corpus, see Classification using movie review corpus in NLTK/Python

That's what exactly I am looking for. The link is very helpful too. Thanks! — user2987, Apr 27 '14 at 22:47

How to read and label line by line a text file using nltk.corpus in Python

1 Answers1