Open a PDF, from a URL, with pdfminer.six

Question

Background: Python 3.7 & pdfminer.six

Using the information found here: Exporting Data from PDFs with Python, I have the following code:

import io

from pdfminer.converter import TextConverter
from pdfminer.pdfinterp import PDFPageInterpreter
from pdfminer.pdfinterp import PDFResourceManager
from pdfminer.pdfpage import PDFPage

def extract_text_from_pdf(pdf_path):
    resource_manager = PDFResourceManager()
    fake_file_handle = io.StringIO()
    converter = TextConverter(resource_manager, fake_file_handle)
    page_interpreter = PDFPageInterpreter(resource_manager, converter)

    with open(pdf_path, 'rb') as fh:
        for page in PDFPage.get_pages(fh, 
                                      caching=True,
                                      check_extractable=True):
            page_interpreter.process_page(page)

        text = fake_file_handle.getvalue()

    # close open handles
    converter.close()
    fake_file_handle.close()

    if text:
        return text

if __name__ == '__main__':
    path = '../_pdfs/mypdf.pdf'
    print(extract_text_from_pdf(path))

This works (yay!), but what I really want to do is request the pdf directly, via its url, rather than open a pdf that has been pre-saved to a local drive.

I have no idea how I need to amend the "with open" logic to call from a remote url, nor am I sure which request library I would be best using for the latest version of Python (requests, urllib, urllib2, etc.?)

I'm new to Python, so please bear that in mind (P.s. I have found other questions on this, but nothing I can make work - possibly because they tend to be quite old.)

Any help would be greatly appreciated! Thank you!

Does this answer your question? [How can i read a PDF file from inline raw\_bytes (not from file)?](https://stackoverflow.com/questions/47177060/how-can-i-read-a-pdf-file-from-inline-raw-bytes-not-from-file) — hellpanderr, Jun 14 '20 at 22:58

score 4 · Accepted Answer · answered Jun 17 '20 at 00:28

I solved it as follows:

from io import StringIO, BytesIO
import urllib.request

from pdfminer.converter import TextConverter
from pdfminer.pdfinterp import PDFPageInterpreter, PDFResourceManager
from pdfminer.pdfpage import PDFPage

def extract_text_from_pdf_url(url, user_agent=None):
    resource_manager = PDFResourceManager()
    fake_file_handle = StringIO()
    converter = TextConverter(resource_manager, fake_file_handle)    

    if user_agent == None:
        user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

    headers = {'User-Agent': user_agent}
    request = urllib.request.Request(url, data=None, headers=headers)

    response = urllib.request.urlopen(request).read()
    fb = BytesIO(response)

    page_interpreter = PDFPageInterpreter(resource_manager, converter)

    for page in PDFPage.get_pages(fb,
                                caching=True,
                                check_extractable=True):
        page_interpreter.process_page(page)


    text = fake_file_handle.getvalue()

    # close open handles
    fb.close()
    converter.close()   
    fake_file_handle.close()

    if text:
        # If document has instances of \xa0 replace them with spaces.
        # NOTE: \xa0 is non-breaking space in Latin1 (ISO 8859-1) & chr(160)
        text = text.replace(u'\xa0', u' ')

        return text

score 0 · Answer 2 · answered Jun 02 '20 at 17:35

You can use PyPDF2 to parse a pdf file.
Try this :

import requests, PyPDF2
# Fill address with your url
try:
    response = requests.get(address)
except:
    print("Error")
my_raw_data = response.content
with open("my_pdf.pdf", 'wb') as my_data:
    my_data.write(my_raw_data)
my_data.close()
open_pdf_file = open("my_pdf.pdf", 'rb')
try:
    read_pdf = PyPDF2.PdfFileReader(open_pdf_file)
except:
    print("Failed to read, Press Enter to continue :")
if read_pdf.isEncrypted:
    read_pdf.decrypt("")
n = read_pdf.getNumPages()
for x in range(0,n):
    try:
        s = read_pdf.getPage(x).extractText()
    except:
        print("Error in",i)
        continue
    print(s)

Thank you, However, PyPDF2 doesn't seem to do a very good job of extracting all the text (items are literally missing), whereas pdfminer.six gets everything I need. — HapiDaze, Jun 02 '20 at 18:02

score 0 · Answer 3 · answered Aug 08 '21 at 01:36

A more minimal solution to retrieve a pdf from a url, in a format that can be used with pdfminer.six is:

def pdf_getter(url:str):
    '''
    retrives pdf from url as bytes object
    '''
    open = urllib.request.urlopen(url).read()
    return io.BytesIO(open)

The PDFParser() function and high level extract_text() function both will accept the returned object as an input.

Open a PDF, from a URL, with pdfminer.six

3 Answers3