Extract depth frame from RealSense camera?

Question

i want to get continuous distance frames from the intel realsense camera. so, i want to generate a Mat object (1 channel) with size 1280*720 (should be the resolution of the 415 realsense cam). this matrix should content only the distance information for every pixel.

In the examples of the realsense sdk you can find the file im-show which creates a Mat object, but this one is a BGR coded colored distance image.

You surely could convert this frame into HSV and getting the distance simply by the H channel. But, i think its not very clean solution to convert first the distance data into BGR, then into HSV and back into distance.

So, i made any changes on the example im-show:

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
// not sure if all of these includes are needed: 
#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <iostream>

using namespace std;
using namespace cv;

int main() 
{
    // Declare RealSense pipeline, encapsulating the actual device and sensors
    rs2::pipeline pipe;
    // Start streaming with default recommended configuration
    pipe.start();

    const auto window_name = "Display Image";
    namedWindow(window_name, WINDOW_AUTOSIZE);

    while (waitKey(1) < 0 && getWindowProperty(window_name, WND_PROP_AUTOSIZE) >= 0)
    {
        rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera
        rs2::frame depth = data.get_depth_frame();

        // Query frame size (width and height)
        const int w = depth.as<rs2::video_frame>().get_width();
        const int h = depth.as<rs2::video_frame>().get_height();

        // Create OpenCV matrix of size (w,h) from the colorized depth data (bgr values)
        Mat temp(Size(w, h), CV_8UC1, (void*)depth.get_data(), Mat::AUTO_STEP);

        //Update the window with new data
        imshow(window_name, temp);
    }

    return EXIT_SUCCESS;
}

i get no errors when compiling it, but the output data is really strange.

any ideas?

score 4 · Answer 1 · edited Dec 18 '19 at 19:34

Thanks, but I found a really simple solution by myself.

// CAPTURE A FRAME
rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera
rs2::depth_frame depth = data.get_depth_frame(); // Get a depth frame 

// Create OpenCV matrix of size (w,h) from the colorized depth data
Mat depth_img(Size(width, height), CV_16UC1, (void*)depth.get_data(), Mat::AUTO_STEP);

// Convert 16bit image to 8bit image
depth_img.convertTo(depth_img, CV_8UC1, 15 / 256.0);

score 0 · Answer 2 · answered Apr 06 '22 at 12:55

an alternative way to @apct solution is:

// Declare depth colorizer for pretty visualization of depth data
rs2::colorizer color_map;
color_map.set_option(RS2_OPTION_COLOR_SCHEME, 2.f); // White to Black
rs2::frame frame_depth = data.get_depth_frame().apply_filter(color_map);
auto vf_depth = frame_depth.as<rs2::video_frame>();
Mat image_depth(Size(w, h), CV_8UC3, (void*)vf_depth.get_data(), Mat::AUTO_STEP);

this approach (of applying a color map, instead of the conversion), gives a more detailed depth information, but with inverted values (bigger values are closer to the sensor, smaller values are far from the sensor).

Extract depth frame from RealSense camera?

2 Answers2