Recognize text in a PDF file rotated at any angle

Question

I have a simple program (code from the documentation of the docTR library) that recognizes text in a pdf file. If the text is perfectly aligned, then there are no problems with text recognition, but if the document is rotated to the right or left, then problems begin with text recognition.

I may receive documents that are not only rotated exactly 90,180 or 270 degrees. Crooked scanned documents can come rotated in any angle (as in the pictures above).

I would like with your help to find a solution that will help me rotate the table / text (or the whole pdf) in my pdf straight, for easy text recognition, as in the picture below.

Perhaps there are already similar solutions, but I have not found them yet. I would be grateful if you point me to existing solutions or help me write code with my own solution.

from doctr.io import DocumentFile
from doctr.models import ocr_predictor

ocr = ocr_predictor(pretrained=True)

doc = DocumentFile.from_pdf("my/path.pdf")
result = ocr(doc)
result.show(doc)

Answer 1

These are my thoughts on the proposed problem:

• If you are scanning the tables from paper, then the document (even if it is pdf format) it contains an image.
• You know that you need to rotate the document for docTR to read it, but from what I read in docTR repository you could also transform the pdf to image and make docTR scan it as an image.
• But, why are you making the pdf into an image? I think it might be easier to do the next two steps if the file is an image:
• First you need to know the angle(amount in degrees, or radians - different for each file) you want to rotate the image. For that, you need to scan the image for "long straight lines" and get their angles (the table borders). You will get many angles, and you only need one, so you might have to get a bit creative there (you could, for example, in the last step scan the file multiple times with docTR for different angles, measuring the success of the result according to the amount of data extracted)
• Once you have your angle(or angles), you rotate the image file to the specific angle you previously calculated
• Last step: use docTR to scan the rotated image

I know this is not a snippet, copy-paste solution. Hopefully you find an easier way to get there. But this would be my approach if nothing easier worked.

Answer 2

You can use OCRmyPDF, which is very good OCR library:

ocrmypdf --rotate-pages input_scanned.pdf

The flag can fix pages that are misrotated. I have no experience with doctr.

Answer 3

You can use a detection model that has been trained on rotated documents and pass the option assume_straight_pages accordingly:

predictor = detection_predictor('db_resnet50_rotation', pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True)

Here is the official documentation.

Answer 4

Before using DocTR for your task. You can use tesseract OCR to rotate your pdf image as per the alignment of the text. The source code and detailed implementation has been provided here: https://pyimagesearch.com/2022/01/31/correcting-text-orientation-with-tesseract-and-python/

Your flow might look as follows:

1. Read pdf and get image.
2. Send image to tesseract ocr for realignment.
3. Send the response to DocTR for character recognition.

Answer 5

Step 1: Convert pdf to image.

Step 2: Read image with opencv

import numpy
import math
import cv2
import matplotlib.pyplot as plt

img = cv2.imread("test.png",0) #grayscale

Step 3: Preprocess if needed. See thresholding etc. (not done cos its not needed for your image example)

Step 4: Use Canny edge detection and Hough lines

dst = cv2.Canny(img, 50, 200, None, 3) #see Canny docs
lines = cv2.HoughLines(dst, 1, np.pi / 180, 150, None, 0, 0) # See docs

Step 5: Convert all your lines angles to degrees and find some best fit.

deg_lines = [round(np.rad2deg(i[0][1]))%90 for i in lines] 
#lines is in format [[rho,theta]]
#we also mod by 90 as the lines should be orthogonal on page. I.E 90degrees
#deg_lines now contains the degree angles of all lines found in the image. 
candidates_angle = round(np.mean(deg_lines)) # or use the median/mode
#candidates_angle now contatins to the nearest degree the current orientation angle of your doc. Rotate it to the correct angle and you should be good. 



cdst = img.copy() #Just to visualize your lines
if lines is not None:
    for i in range(0, len(lines)):
        rho = lines[i][0][0]
        theta = lines[i][0][1]
        a = math.cos(theta)
        b = math.sin(theta)
        x0 = a * rho
        y0 = b * rho
        pt1 = (int(x0 + 1000*(-b)), int(y0 + 1000*(a)))
        pt2 = (int(x0 - 1000*(-b)), int(y0 - 1000*(a)))
        cv2.line(cdst, pt1, pt2, (0,0,255), 3, cv2.LINE_AA)
plt.imshow(cdst)
plt.show()

Step 6: Rotate your image and run your code. Or refer to cv2/PIL libraries for rotating an image you did this already so it should work.

Additional docs:

https://docs.opencv.org/3.4/d9/db0/tutorial_hough_lines.html

https://docs.opencv.org/3.4/da/d22/tutorial_py_canny.html

https://docs.opencv.org/4.x/d7/d4d/tutorial_py_thresholding.html

Please let me know if you have any additional questions.