Python (nltk) - UnicodeDecodeError: 'ascii' codec can't decode byte

Question

I'm new to NLTK. I'm getting this error and I've searched around for encoding/decoding and specifically the UnicodeDecodeError but this error seems specific to the NLTK source code.

Here's the error:

Traceback (most recent call last):
  File "A:\Python\Projects\Test\main.py", line 2, in <module>
    print(pos_tag(word_tokenize("John's big idea isn't all that bad.")))
  File "A:\Python\Python\lib\site-packages\nltk\tag\__init__.py", line 100, in pos_tag
    tagger = load(_POS_TAGGER)
  File "A:\Python\Python\lib\site-packages\nltk\data.py", line 779, in load
    resource_val = pickle.load(opened_resource)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcb in position 0: ordinal not in range(128)

How do I go around fixing this error?

Here's what causes the error:

from nltk import pos_tag, word_tokenize
print(pos_tag(word_tokenize("John's big idea isn't all that bad.")))

There's nothing in the code you show here that would generate the error. Print the `repr` of the string you're passing. — Mark Ransom, Aug 25 '14 at 21:08
@MarkRansom I don't know what you mean, the function `pos_tag` is causing the error. I think the encoding error is generated on the pickle.load function. I'm not sure what to do. — user3422952, Aug 25 '14 at 21:22

score 5 · Answer 1 · answered Jan 14 '15 at 17:06

5

try this... NLTK 3.0.1 with Python 2.7.x

import io
f = io.open(txtFile, 'rU', encoding='utf-8')

answered Jan 14 '15 at 17:06

Simone Dagli Orti

476
6
5

Worked like a charm! I use nltk 3.1 and Python 2.7.x. – Muffintop Oct 27 '15 at 19:12
This is great ! can you also explain why using io solves the problem ? – minocha Sep 01 '16 at 22:55

LuckyMatina · Answer 2 · 2015-03-04T19:08:53.120

4

I had the same problem with you. I use Python 3.4 in Windows 7.

I had installed the "nltk-3.0.0.win32.exe" (from here). But when i installed the "nltk-3.0a4.win32.exe" (from here), my problem with nltk.pos_tag was solved. Check it.

EDIT: If the second link doesn't work, you can look here.

edited Mar 04 '15 at 19:08

answered Sep 28 '14 at 17:39

LuckyMatina

41
6

the second link seems to be broken. Do you have any alternate links? – Arnab Chakraborty Jan 03 '15 at 10:29

score -2 · Answer 3 · edited May 23 '17 at 12:00

-2

Duplicate: NLTK 3 POS_TAG throws UnicodeDecodeError

Long story short: NLTK isn't compatible with Python 3. You have to use NLTK 3 which sounds a bit experimental at this point.

edited May 23 '17 at 12:00

Community

1
1

answered Sep 03 '14 at 02:22

Dave

1

4

I am using NLTK 3 and Python 3.4 and still get this error. – Gustavo Puma Sep 06 '14 at 01:26

score -2 · Answer 4 · answered Sep 05 '14 at 09:36

-2

Try using the module "textclean"

>>> pip install textclean

Python code

from textclean.textclean import textclean
text = textclean.clean("John's big idea isn't all that bad.")
print pos_tag(word_tokenize(text))

answered Sep 05 '14 at 09:36

Shivamshaz

262
2
3
10

this module sounds like a _horrible_ idea. it's particularly bad here, because the error is occurring when trying to decode a pickle — a structured data format that you will irreparably destroy if you try to blindly "clean" it. – Eevee Dec 18 '14 at 01:32

Python (nltk) - UnicodeDecodeError: 'ascii' codec can't decode byte

4 Answers4

Linked

Related