Replace (cid:) with chars using Python when extracting text from PDF files

Question

I wrote a code in Python that extracts text from PDF files. But for some files Im getting some strange output. This is my code:

import requests

from io import BytesIO
from pdfminer.high_level import extract_text, extract_pages

pdf_link = 'https://www.neerach.ch/public/upload/assets/1417/MTB0321.pdf'

response = requests.get(pdf_link)
with BytesIO(response.content) as data:
        
    num_of_pages = len(list(extract_pages(data)))
    print('number of pages', num_of_pages)

    #extract first 5 pages
    text = extract_text(data, password='', page_numbers = None, maxpages = 5, caching=True, codec='utf-8', laparams=None)
    text = str(text)
    text = text.replace('\n\n\n', '\n\n').strip()
    print(text)

The result that im getting:

cid:3)
(cid:3)
(cid:3)
(cid:3)

(cid:3)
(cid:3)
(cid:3)

Nr. 3 | 2021

März 2021

(cid:3)
(cid:57)(cid:72)(cid:85)(cid:75)(cid:68)(cid:81)(cid:71)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:71)(cid:72)(cid:86)(cid:3)(cid:42)(cid:72)(cid:80)(cid:72)(cid:76)(cid:81)(cid:71)(cid:72)(cid:85)(cid:68)(cid:87)(cid:72)(cid:86)(cid:3)
(cid:3)
(cid:54)(cid:70)(cid:75)(cid:88)(cid:79)(cid:72)(cid:81)(cid:3)
(cid:3)
(cid:54)(cid:82)(cid:93)(cid:76)(cid:68)(cid:79)(cid:72)(cid:3)(cid:39)(cid:76)(cid:72)(cid:81)(cid:86)(cid:87)(cid:72)(cid:3)
(cid:3)
(cid:48)(cid:76)(cid:87)(cid:87)(cid:72)(cid:76)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:39)(cid:82)(cid:85)(cid:73)(cid:89)(cid:72)(cid:85)(cid:72)(cid:76)(cid:81)(cid:72)(cid:3)
(cid:3)
(cid:48)(cid:76)(cid:87)(cid:87)(cid:72)(cid:76)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:68)(cid:88)(cid:86)(cid:90)(cid:108)(cid:85)(cid:87)(cid:76)(cid:74)(cid:72)(cid:85)(cid:3)(cid:57)(cid:72)(cid:85)(cid:72)(cid:76)(cid:81)(cid:72)(cid:3)
(cid:3)
(cid:48)(cid:76)(cid:87)(cid:87)(cid:72)(cid:76)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:46)(cid:76)(cid:85)(cid:70)(cid:75)(cid:74)(cid:72)(cid:80)(cid:72)(cid:76)(cid:81)(cid:71)(cid:72)(cid:81)(cid:3)

(cid:20)(cid:3)

(cid:23)(cid:3)

(cid:20)(cid:21)(cid:3)

(cid:21)(cid:20)(cid:3)

(cid:21)(cid:24)(cid:3)

Mitteilungsblatt Neerach | Gemeindeverwaltung Neerach | Binzmühlestrasse 14 | 8173 Neerach
044 859 16 16 | einwohnerkontrolle@neerach.ch | www.neerach.ch

(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)
(cid:3)
(cid:3)
(cid:3)

Basically each char is replaced with (cid:<number>). But also, as you can see, I have some string values. I have also tried different types of encoding such as:

encodings = ["Adobe-GB1-0", "Adobe-GB1-1", "Adobe-GB1-2", "Adobe-GB1-3", "Adobe-GB1-4", "Adobe-GB1-5", "GB-EUC-H", "GB-EUC-V", "GB-H", "GB-V", "GBK-EUC-H", "GBK-EUC-V", "GBK2K-H", "GBK2K-V", 'utf-8',
           "GBKp-EUC-H", "GBKp-EUC-V", "GBT-EUC-H", "GBT-EUC-V", "GBT-H", "GBT-V", "GBTpc-EUC-H", "GBTpc-EUC-V", "GBpc-EUC-H", "GBpc-EUC-V", "UniGB-UCS2-H", "UniGB-UCS2-V", "UniGB-UTF16-H", 'utf-16',
          "UniGB-UTF16-V", "UniGB-UTF32-H", "UniGB-UTF32-V", "UniGB-UTF8-H", "UniGB-UTF8-V", "78-EUC-V", "78-H", "78-RKSJ-H", "78-RKSJ-V", "78-V", "78ms-RKSJ-H", "78ms-RKSJ-V", "83pv-RKSJ-H", 'utf-32',
          "90ms-RKSJ-H", "90ms-RKSJ-V", "90msp-RKSJ-H", "90msp-RKSJ-V", "90pv-RKSJ-H", "90pv-RKSJ-V", "Add-H", "Add-RKSJ-H", "Add-RKSJ-V", "Add-V", "Adobe-Japan1-0", "Adobe-Japan1-1", "Adobe-Japan1-2",
          "Adobe-Japan1-3", "Adobe-Japan1-4", "Adobe-Japan1-5", "Adobe-Japan1-6", "Adobe-Japan1-7", "EUC-H", "EUC-V", "Ext-H", "Ext-RKSJ-H", "Ext-RKSJ-V", "Ext-V", "H", "Hankaku", "Hiragana",  "HKm471-B5-H",
          "Adobe-KR-9", "UniAKR-UTF16-H", "UniAKR-UTF32-H", "UniAKR-UTF8-H", "ETenms-B5-V", "HKdla-B5-H", "HKdla-B5-V", "HKdlb-B5-H", "HKdlb-B5-V", "HKgccs-B5-H", "HKgccs-B5-V", "HKm314-B5-H", "HKm314-B5-V"]

How can I encode this type of response? What should I add/change in my code in order to get text resonse?

Does this answer your question? [What to do with CIDs in text extracted by PDFMiner?](https://stackoverflow.com/questions/50773909/what-to-do-with-cids-in-text-extracted-by-pdfminer). The problem is caused by fonts. I suggest you used PyPDF to extract the data. — Ceres, Mar 16 '21 at 13:46
As a first test open the PDF in Adobe Acrobat Reader, copy all text, and paste it into an editor. If that process does not give you the desired text, the PDF doesn't contain the information required for regular text extraction, so normal text extractors will fail, too. In case of your file you'll see some parts extracted all right and some extracted as garbled data. So, don't expect to fare better than that with pdfminer. — mkl, Mar 16 '21 at 13:49
Here's my artless approach: `x='(cid:57)(cid:72)(cid:85)(cid:75)(cid:68)(cid:81)(cid:71)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)'; y = x.replace(')(cid:',',').replace('(cid:','').replace(')','').split(','); ''.join([chr(int(i)+29) for i in y])` returns `'Verhandlungen'` — JosefZ, Mar 16 '21 at 16:04
and please, I have updated the question with bigger output example, how would you handle that? — taga, Mar 16 '21 at 17:37
Unfortunately there is not an easy solution for these *cid:*. @JosefZ solution works for his example, but if you apply his answer across your PDFs it will not work properly. For example, one of your PDFs outputs *cid:1*. JosefZ solution coverts this to a space, but the character in the document is a checkmark, which is used as a bullet point. I have been looking at a way to convert your PDFs to a format that might be able to be read by the PDF extractors, but so far all efforts have failed. — Life is complex, Mar 17 '21 at 14:44
After doing some more research into @JosefZ solution it could potentially work with some caveats. These caveats are linked to certain characters that cannot be easily converted, such as checkmarks. I would likely recommend creating a solution that can extract content without *cid:* and another that would process content with lots of *cid:*. This solution will require testing and tweaking. — Life is complex, Mar 17 '21 at 15:19

score 9 · Accepted Answer · answered Mar 17 '21 at 17:55

With more caveats (some of them told in Life is complex's comments) and using your sample data:

import re
def cidToChar(cidx):
    return chr(int(re.findall(r'\(cid\:(\d+)\)',cidx)[0]) + 29)

xx = '''
(cid:3)
(cid:3)
(cid:3)
(cid:3)

(cid:3)
(cid:3)
(cid:3)

Nr. 3 | 2021

März 2021

(cid:3)
(cid:57)(cid:72)(cid:85)(cid:75)(cid:68)(cid:81)(cid:71)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:71)(cid:72)(cid:86)(cid:3)(cid:42)(cid:72)(cid:80)(cid:72)(cid:76)(cid:81)(cid:71)(cid:72)(cid:85)(cid:68)(cid:87)(cid:72)(cid:86)(cid:3)
(cid:3)
(cid:54)(cid:70)(cid:75)(cid:88)(cid:79)(cid:72)(cid:81)(cid:3)
(cid:3)
(cid:54)(cid:82)(cid:93)(cid:76)(cid:68)(cid:79)(cid:72)(cid:3)(cid:39)(cid:76)(cid:72)(cid:81)(cid:86)(cid:87)(cid:72)(cid:3)
(cid:3)
(cid:48)(cid:76)(cid:87)(cid:87)(cid:72)(cid:76)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:39)(cid:82)(cid:85)(cid:73)(cid:89)(cid:72)(cid:85)(cid:72)(cid:76)(cid:81)(cid:72)(cid:3)
(cid:3)
(cid:48)(cid:76)(cid:87)(cid:87)(cid:72)(cid:76)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:68)(cid:88)(cid:86)(cid:90)(cid:108)(cid:85)(cid:87)(cid:76)(cid:74)(cid:72)(cid:85)(cid:3)(cid:57)(cid:72)(cid:85)(cid:72)(cid:76)(cid:81)(cid:72)(cid:3)
(cid:3)
(cid:48)(cid:76)(cid:87)(cid:87)(cid:72)(cid:76)(cid:79)(cid:88)(cid:81)(cid:74)(cid:72)(cid:81)(cid:3)(cid:46)(cid:76)(cid:85)(cid:70)(cid:75)(cid:74)(cid:72)(cid:80)(cid:72)(cid:76)(cid:81)(cid:71)(cid:72)(cid:81)(cid:3)

(cid:20)(cid:3)

(cid:23)(cid:3)

(cid:20)(cid:21)(cid:3)

(cid:21)(cid:20)(cid:3)

(cid:21)(cid:24)(cid:3)

Mitteilungsblatt Neerach | Gemeindeverwaltung Neerach | Binzmühlestrasse 14 | 8173 Neerach
044 859 16 16 | einwohnerkontrolle@neerach.ch | www.neerach.ch

(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)
(cid:3)
(cid:3)
(cid:3)
'''
for x in xx.split('\n'):
  if x != '' and x != '(cid:3)':         # merely to compact the output
    abc = re.findall(r'\(cid\:\d+\)',x)
    if len(abc) > 0:
        for cid in abc: x=x.replace(cid, cidToChar(cid))
    print(repr(x).strip("'"))

Output shows that cidToChar algorithm fails e.g. for German letters with umlaut/diaeresis (e.g. ausw\x89rtiger instead of auswärtiger): .\SO\66656067.py

Nr. 3 | 2021
März 2021
Verhandlungen des Gemeinderates
Schulen
Soziale Dienste
Mitteilungen Dorfvereine
Mitteilungen ausw\x89rtiger Vereine
Mitteilungen Kirchgemeinden
1
4
12
21
25
Mitteilungsblatt Neerach | Gemeindeverwaltung Neerach | Binzmühlestrasse 14 | 8173 Neerach
044 859 16 16 | einwohnerkontrolle@neerach.ch | www.neerach.ch

This was a solid way to solve this problem, so it gets my upvote. — Life is complex, Mar 17 '21 at 20:13

Replace (cid:) with chars using Python when extracting text from PDF files

1 Answers1

Linked