low quality image with cv2.warperspective

Question

I am working with the perspective transformation using opencv on python. So I have found the homography matrix between two sets of points using SIFT detector. For now, I would like to transform an image using the cv2.warperspective function but it turns out the quality of the warped image is very low.

I did some google search and found this, Image loses quality with cv2.warpPerspective. So by using the method from the accepted answer which try with different interpolation flags for the warperspective but it didn't help. Please do suggest what to do to overcome the issue. Thanks in advance.

Code:

import cv2
import numpy as np
import matplotlib.pyplot as plt
import time
import copy

def main():
    picture = cv2.imread("src/paper.png")
    cap = cv2.VideoCapture("src/video0.mp4")
    query = cv2.imread("src/qr.png")

    while True:
        _, frame = cap.read()
        h,w = frame.shape[:2]
        rgb = cv2.cvtColor(query, cv2.COLOR_BGR2RGB)
        paper = cv2.resize(paper, (query.shape[1], query.shape[0]), interpolation = cv2.INTER_AREA)

        # declare sift object detector
        sift = cv2.SIFT_create()

        # point and object detection
        query_kp, query_des = sift.detectAndCompute(rgb,None)
        train_kp, train_des = sift.detectAndCompute(frame.astype(np.uint8),None)

        match_points = []
        bf = cv2.BFMatcher(cv2.NORM_L2,crossCheck=False)
        match = bf.knnMatch(query_des,train_des,k=2)
        # ratio test
        ratio = 0.8
        for ptn1,ptn2 in match:
            if ptn1.distance < ratio * ptn2.distance:
                match_points.append(ptn1)

        # at least 4 points for projection
        if len(match_points) >= 4:
            set_point1 = np.float32([query_kp[ptn.queryIdx].pt for ptn in match_points]).reshape((-1, 1, 2))
            set_point2 = np.float32([train_kp[ptn.trainIdx].pt for ptn in match_points]).reshape((-1, 1, 2))
            frame = projection(frame, paper, set_point1, set_point2)

        cv2.imshow("frame", frame)
        cv2.waitKey(1)


def projection(frame, picture, set_point1, set_point2):
    h,w = frame.shape[:2]
    homography_matrix, _ = cv2.findHomography(set_point1,set_point2,cv2.RANSAC,5)
    warpedImg = cv2.warpPerspective(picture, homography_matrix, (w,h), flags=cv2.WARP_FILL_OUTLIERS)

    # overlay the warpedImg to original frame
    x,y,z = np.where(warpedImg != [0,0,0])
    frame[x,y,z] = warpedImg[x,y,z]
    return frame

main()

Example output - The image under the profile picture section (I understand due to the aspect ratio so it looks small but the quality is just bad and the words in the warped image are all kinda distorted):

Answer 1

did you test various interpolation methods, supported in OpenCV? I reckon that the results of bilinear or bicubic interpolation (i.e., INTER_LINEAR or INTER_CUBIC) would be surprising for you. You can change your code as below (indeed you are are using INTER_NEAREST which can distort image contents):

warpedImg = cv2.warpPerspective(picture, homography_matrix, (w,h), flags= cv2.WARP_FILL_OUTLIERS + cv2.INTER_LINEAR)

From OpenCV source:

enum InterpolationFlags{
    /** nearest neighbor interpolation */
    INTER_NEAREST        = 0,
    /** bilinear interpolation */
    INTER_LINEAR         = 1,
    /** bicubic interpolation */
    INTER_CUBIC          = 2,
    /** resampling using pixel area relation. It may be a preferred method for image decimation, as
    it gives moire'-free results. But when the image is zoomed, it is similar to the INTER_NEAREST
    method. */
    INTER_AREA           = 3,
    /** Lanczos interpolation over 8x8 neighborhood */
    INTER_LANCZOS4       = 4,
    /** Bit exact bilinear interpolation */
    INTER_LINEAR_EXACT = 5,
    /** Bit exact nearest neighbor interpolation. This will produce same results as
    the nearest neighbor method in PIL, scikit-image or Matlab. */
    INTER_NEAREST_EXACT  = 6,
    /** mask for interpolation codes */
    INTER_MAX            = 7,
    /** flag, fills all of the destination image pixels. If some of them correspond to outliers in the
    source image, they are set to zero */
    WARP_FILL_OUTLIERS   = 8,
    /** flag, inverse transformation

    For example, #linearPolar or #logPolar transforms:
    - flag is __not__ set: \f$dst( \rho , \phi ) = src(x,y)\f$
    - flag is set: \f$dst(x,y) = src( \rho , \phi )\f$
    */
    WARP_INVERSE_MAP     = 16
};