OpenCV warping image based on calcOpticalFlowFarneback

Question

I'm trying to perform a complex warp of an image using Dense Optical Flow. I am trying to warp the second image into roughly the same shape as the first image.

cv::Mat flow;
cv::calcOpticalFlowFarneback( mGrayFrame1, mGrayFrame2, flow, 0.5, 3, 15, 3, 5, 1.2, 0 );

cv::Mat newFrame = cv::Mat::zeros( frame.rows, frame.cols, frame.type() );
cv:remap( frame, newFrame, flow, cv::Mat(), CV_INTER_LINEAR );

I calculate the flow from two grayscale frames. I am now trying to remap my original (i.e. non-grayscale) image using this flow information using the cv::remap function. However, I get a very badly distorted image from it. I simply end up with an orange and black image that bears a small resemblance to my original image.

How do I use cv::remap with the calculated flow?

Hello, I saw this post and I have approximately the same problem (exactly the same problem in definitive) and I was wondering if you found a way to do it? (I know this post is old, and i'm sorry for that, but I'm not finding any response in actual time :o) Thanks a lot! — Raph Schim, Apr 25 '16 at 14:26
@RaphaelSchimchowitsch: Probably best to start your own question. I'm not if I even still have the code I wrote to do this ... I did get something working though. Now sure if it was with optical flow though ... — Goz, Apr 25 '16 at 15:33
Ahah, ok, I already posted my own question but I found this post so I was just wondering if you could help me :) But ok ^^ Thanks a lot for your response :) — Raph Schim, Apr 26 '16 at 06:33

score 18 · Accepted Answer · edited Mar 12 '21 at 01:32

The remap function cannot work with flow directly. One must use a separate map that is computed by taking the backwards flow (from frame2 to frame1) and then offsetting each flow vector by its (x, y) location on the pixel grid. See details below.

Recall the backwards optical flow formula:

frame1(x, y) = frame2(x + flowx(x, y), y + flowy(x, y))

The remap function transforms the source image using a specified map:

dst(x, y) = src(mapx(x, y), mapy(x, y))

Comparing the two equations above, we may determine the map that remap requires:

mapx(x, y) = x + flowx(x, y)
mapy(x, y) = y + flowy(x, y)

Example:

Mat flow; // backward flow
calcOpticalFlowFarneback(nextFrame, prevFrame, flow);

Mat map(flow.size(), CV_32FC2);
for (int y = 0; y < map.rows; ++y)
{
    for (int x = 0; x < map.cols; ++x)
    {
        Point2f f = flow.at<Point2f>(y, x);
        map.at<Point2f>(y, x) = Point2f(x + f.x, y + f.y);
    }
}

Mat newFrame;
remap(prevFrame, newFrame, map);

Brilliant, that explains a lot. Unfortunately the flow doesn't contain the data I thought it did so back to the drawing board and probably another question soon! Thanks! — Goz, Jul 04 '13 at 06:53
Note the use of backward flow, i.e. from `nextFrame` to `prevFrame`. This is needed because the flow vectors are given for source pixels ("where did this go"), while we are interested in destination pixels ("where did this come from"). — lapis, Aug 26 '15 at 10:38

Yang · Answer 2 · 2023-08-17T09:33:52.497

Here is a Python solution to warp image from optical flow:

import cv2
import numpy as np

def warp_flow(flow, img1=None, img2=None, interpolation=cv2.INTER_LINEAR):
    """Use remap to warp flow, generating a new image. 
Args:
    flow (np.ndarray): flow
    img1 (np.ndarray, optional): previous frame
    img2 (np.ndarray, optional): next frame
Returns:
    warped image
If img1 is input, the output will be img2_warped, but there will be multiple pixels corresponding to a single pixel, resulting in sparse holes. 
If img2 is input, the output will be img1_warped, and there will be no sparse holes. The latter approach is preferred.
    """
    h, w, _ = flow.shape
    remap_flow = flow.transpose(2, 0, 1)
    remap_xy = np.float32(np.mgrid[:h, :w][::-1])
    if img1 is not None:
        uv_new = (remap_xy + remap_flow).round().astype(np.int32)
        mask = (uv_new[0] >= 0) & (uv_new[1] >= 0) & (uv_new[0] < w) & (uv_new[1] < h)
        uv_new_ = uv_new[:, mask]
        remap_xy[:, uv_new_[1], uv_new_[0]] = remap_xy[:, mask]
        remap_x, remap_y = remap_xy
        img2_warped = cv2.remap(img1, remap_x, remap_y, interpolation)
        mask_remaped = np.zeros((h, w), np.bool8)
        mask_remaped[uv_new_[1], uv_new_[0]] = True
        img2_warped[~mask_remaped] = 0
        return img2_warped
    elif img2 is not None:
        remap_x, remap_y = np.float32(remap_xy + remap_flow)
        return cv2.remap(img2, remap_x, remap_y, interpolation)


img1 = cv2.imread("img1.jpg")
img2 = cv2.imread("img2.jpg")

flow = cv2.calcOpticalFlowFarneback(
    img1.mean(-1), img2.mean(-1), None, 0.5, 3, 15, 3, 5, 1.2, 0
)

img2_warped = warp_flow(flow, img1=img1)
img1_warped = warp_flow(flow, img2=img2)

cv2.imwrite("warped.jpg", np.vstack([img1_warped, img2_warped]))
cv2.imwrite("target.jpg", np.vstack([img1, img2]))

The examples img1, img2 and flow visualization :

The results, left: warped.jpg,right: target.jpg:

OpenCV warping image based on calcOpticalFlowFarneback

2 Answers2

Linked