Python: special characters giving me problems (from PDFminer)

Question

I used pdf2text from PDFminer to reduce a PDF to text. Unfortunately it contains special characters. Let me show output from my console

>>>a=pdf_to_text("ap.pdf")

heres a sample of it, a little truncated

>>>a[5000:5500]
'f one architect. Decades ...... but to re\xef\xac\x82ect\none set of design ideas, than to have one that contains many\ngood but independent and uncoordinated ideas.\n1 Joshua Bloch, \xe2\x80\x9cHow to Design a Good API and Why It Matters\xe2\x80\x9d, G......=-3733'

I understood that I must encode it

>>>a[5000:5500].encode('utf-8')
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 237: ordinal not in range(128)

I searched around a bit and tried them, notably Replace special characters in python . The input comes from PDFminer, so its tough (AFAIK) to control that. What is the way to make proper plaintext from this output?

What am I doing wrong?

--A quick fix: change PDFminer's codec to ascii- but it's not a lasting solution--

--Abandoned the quick fix for the answer- changing the codec removes information --

--A relavent topic as mentioned by Maxim http://en.wikipedia.org/wiki/Windows-1251 --

Thanks for this question! Im beginner in Python could you maybe post a demo code how to use Pdfminer so that this errors dont appaer? Thanks — John Smith, Nov 11 '14 at 18:07

score 12 · Accepted Answer · edited Oct 03 '12 at 23:07

12

This problem often occurs when non-ASCII text is stored in str objects. What you are trying to do is to encode in utf-8 a string already encoded in some encoding (because it contains characters with codes above 0x7f).

To encode such a string in utf-8 it has to be first decoded. Assuming that the original text encoding is cp1251 (replace it with your actual encoding), something like the following would do the trick:

u = s.decode('cp1251')  # decode from cp1251 byte (str) string to unicode string
s = u.encode('utf-8')   # re-encode unicode string to  utf-8 byte (str) string

Basically, the above snippet does what iconv --from-code=CP1251 --to-code=UTF-8 command does, i.e. it converts the string from one encoding to another.

Some useful links:

edited Oct 03 '12 at 23:07

Pedro Romano

10,973
4
46
50

answered Jul 29 '11 at 13:06

Maxim Egorushkin

131,725
17
180
271

Yes, this works close to flawlessly! I got a few artifacts like "...Ð²Ð‚ÑšDelivering Custo.." but it was a PDF made by amateurs for maximum glitz. Cleaner PDFs are cleanly parsed. – Jesvin Jose Jul 29 '11 at 14:06
Nice one, you need to know your input encoding though. – Maxim Egorushkin Jul 29 '11 at 14:07
I'd definitely write it as `a.decode('cp1250').encode('utf-8')`. – Jan Hudec Aug 12 '11 at 12:48
@JanHudec I spelled it out in two lines to be able to put comments, i.e. for the clarity of exposition only. – Maxim Egorushkin Jul 22 '14 at 19:58
@aitchnyu What IS the PDFMiner encoding - cp1250? – jtlz2 Jul 27 '17 at 12:33

Python: special characters giving me problems (from PDFminer)

1 Answers1

Linked