51

Are there any open source libraries that support table identification & extraction?

By this I mean:

  1. Identify a table structure exists
  2. Classify the table from its contents
  3. Extract data from the table in a useful output format e.g. JSON / CSV etc.

I have looked through similar questions on this topic and found the following:

  • PDFMiner which addresses problem 3, but it seems the user is required to specify to PDFMiner where a table structure exists for each table (correct me if I'm wrong)
  • pdf-table-extract which attempts to address problem 1 but according to the To-Do list, cannot currently identify tables that are separated by whitespace. This is a problem as all tables in my PDFs are separated by whitespace!

Currently, I am thinking that I would have to spend a lot of time developing a Machine Learning solution to identify table structures from PDFs. Therefore, any alternative approaches would be more than welcome!

Trenton McKinney
  • 56,955
  • 33
  • 144
  • 158
Alexander McFarlane
  • 10,643
  • 9
  • 59
  • 100
  • 5
    If you can use tools beyond python, too, you might want to take a look at [tabula](http://tabula.technology/). – mkl Feb 16 '15 at 07:14
  • thanks. Will definitely look into that. I'm keen on finding a solution in python though because of the speed in which python can be written – Alexander McFarlane Feb 16 '15 at 22:09
  • @Alexander McFarlane: Try SLICEmyPDF in 1 of the answers at https://stackoverflow.com/questions/56017702/how-to-extract-table-from-pdf-in-python/72414309#72414309 – 123456 May 29 '22 at 10:20
  • See https://stackoverflow.com/q/17591426/562769 – Martin Thoma Feb 11 '23 at 10:32

3 Answers3

49

After many fruitful hours of exploring OCR libraries, bounding boxes and clustering algorithms - I found a solution so simple it makes you want to cry!

I hope you are using Linux;

pdftotext -layout NAME_OF_PDF.pdf

AMAZING!!

Now you have a nice text file with all the information lined up in nice columns, now it is trivial to format into a csv etc..

It is for times like this that I love Linux, these guys came up with AMAZING solutions to everything, and put it there for FREE!

Ike
  • 1,039
  • 11
  • 10
  • 4
    I was able to get `pdftotext` on Windows 10. Just download the [XPDFTools][1] for Windows. [1]: http://www.xpdfreader.com/download.html – avg Feb 16 '18 at 11:49
  • @avg just got back to it today... basically I'm taking the text file generated by `pdftotext`, splitting it into a list of strings (one for each line), creating a set of field names using `csv.DictWriter()`, and then looping through each line, slicing it into the fields I want and then feeding those back to DictWriter. HTH. https://gist.github.com/memilanuk/c6e0bb9f98076a172d4f39d044ed6ecf – memilanuk Feb 19 '18 at 21:05
  • This library is written for Python 2.x version and doesn't work's with Python 3.x. – Laveena Feb 26 '19 at 16:47
41

You should definitely have a look at this answer of mine:

and also have a look at all the links included therein.

Tabula/TabulaPDF is currently the best table extraction tool that is available for PDF scraping.

Community
  • 1
  • 1
Kurt Pfeifle
  • 86,724
  • 23
  • 248
  • 345
  • 2
    just an update on the effectiveness of this answer... I hacked a solution together using `tabula` last year to iterate through about 100 PDFs that had a few formats in common. It wasn't pretty but it was the best of the worst and saved significant time. – Alexander McFarlane Apr 22 '16 at 21:34
  • Will https://pypi.python.org/pypi/pdftable satisfy the requirements? – N M Sep 20 '17 at 09:56
  • it just works on textbased pdfs and not on images is there anything similiar to this where it can extract data from pdf images ? – Sundeep Pidugu Nov 30 '18 at 06:06
  • @Sundeep: ***Of course*** it can only work on text-based PDFs. If you want to extract tables from an image, you have to attempt running a process of OCR (optical character recognition) on the image first and then apply the table extraction on the text. Final result quality will largely depend on success of the OCR step. There is nothing which would be able to extract tables (or texts) directly from image-only PDFs. – Kurt Pfeifle Nov 30 '18 at 08:57
  • Iam looking for tools that can do that btw thanks for the info @KurtPfeifle – Sundeep Pidugu Nov 30 '18 at 09:07
  • @Sundeep: You could start looking which tools are mentioned here: https://stackoverflow.com/questions/tagged/ocr – Kurt Pfeifle Nov 30 '18 at 14:23
18

I'd just like to add to the very helpful answer from Kurt Pfeifle - there is now a Python wrapper for Tabula, and this seems to work very well so far: https://github.com/chezou/tabula-py

This will convert your PDF table to a Pandas data frame. You can also set the area in x,y co-ordinates which is obviously very handy for irregular data.

Ricky McMaster
  • 4,289
  • 2
  • 24
  • 23