How to project IR image on a 2D plane using OpenCV and PCL

Question

I have a Kinect and I'm using OpenCV and point cloud library. I would like to project the IR Image onto a 2D plane for forklift pallet detection. How would I do that?

I'm trying to detect the pallet in the forklift here is an image:

Answer 1

Where are the RGB data? You can use it to help with the detection. You do not need to project the image onto any plane to detect a pellet. There are basically 2 ways used for detection

1. non-deterministic based on neural network, fuzzy logic, machine learning, etc

   This approach need a training dataset to recognize the object. Much experience is needed for proper training set and classifier architecture/topology selection. But other then that you do not need to program it... as usually some readily available lib/tool is used just configure and pass the data.

2. deterministic based on distance or correlation coefficients

   I would start with detecting specific features like:

   • pallet has specific size
   • pallet has sharp edges and specific geometry shape in depth data
   • pallet has specific range of colors (yellowish wood +/- lighting and dirt)
   • Wood has specific texture patterns

   So compute some coefficient for each feature how close the object is to real pallet. And then just treshold the distance of all coefficients combined (possibly weighted as some features are more robust).

I do not use the #1 approach so I would go for #2. So combine the RGB and depth data (they have to be matched exactly). Then segmentate the image (based on depth and color). After that for each found object classify if it is pallet ...

[Edit1]

Your colored image does not correspond to depth data. The aligned gray-scale has poor quality and the depth data image is also very poor. Is the depth data processed somehow (loosing precision)? If you look at your data from different sides:

You can see how poor it is so I doubt you can use depth data for detection at all...

PS. I used my Align already captured rgb and depth images for the visualization.

The only thing left is the colored image and detect areas with matching color only. Then detect the features and classify. The color of your pallet in the image is almost white. Here HSV reduced colors to basic 16 colors (too lazy to segmentate)

You should obtain range of colors of the pallets possible by your setup to ease up the detection. Then check those objects for the features like size, shape,area,circumference...

[Edit2]

So I would start with Image preprocessing:

1. convert to HSV

2. treshold only pixels close to pallet color

   I chose (H=40,S=18,V>100) as a pallet color. My HSV ranges are <0,255> per channel so Hue angle difference can be only <-180deg,+180deg> max which corresponds to <-128,+128> in my ranges.

3. remove too thin areas

   Just scan all Horizontal an Vertical lines count consequent set pixels and if too small size recolor them to black...

This is the result:

On the left the original image (downsized so it fits to this page), In the middle is the color treshold result and last is the filtering out of small areas. You can play with tresholds and pallet color to change behavior to suite your needs.

Here C++ code:

int tr_d=10;    // min size of pallet [pixels[
int h,s,v,x,y,xx;
color c;

pic1=pic0;
pic1.pf=_pf_rgba;
pic2.resize(pic1.xs*3,pic1.ys); xx=0;
pic2.bmp->Canvas->Draw(xx,0,pic0.bmp); xx+=pic1.xs;

// [color selection]
for (y=0;y<pic1.ys;y++)
 for (x=0;x<pic1.xs;x++)
    {
    // get color from image
    c=pic0.p[y][x];
    rgb2hsv(c);
    // distance to white-yellowish color in HSV (H=40,S=18,V>100)
    h=c.db[picture::_h]-40;
    s=c.db[picture::_s]-18;
    v=c.db[picture::_v];
    // hue is cyclic angular so use only shorter angle
    if (h<-128) h+=256;
    if (h>+128) h-=256;
    // abs value
    if (h<   0) h=-h;
    if (s<   0) s=-s;
    // treshold close colors
    c.dd=0;
    if (h<25)
     if (s<25)
      if (v>100)
       c.dd=0x00FFFFFF;
    pic1.p[y][x]=c;
    }
pic2.bmp->Canvas->Draw(xx,0,pic1.bmp); xx+=pic1.xs;

// [remove too thin areas]
for (y=0;y<pic1.ys;y++)
 for (x=0;x<pic1.xs;)
    {
    for (   ;x<pic1.xs;x++) if ( pic1.p[y][x].dd) break;    // find set pixel
    for (h=x;x<pic1.xs;x++) if (!pic1.p[y][x].dd) break;    // find unset pixel
    if (x-h<tr_d) for (;h<x;h++) pic1.p[y][h].dd=0;         // if too small size recolor to zero
    }
for (x=0;x<pic1.xs;x++)
 for (y=0;y<pic1.ys;)
    {
    for (   ;y<pic1.ys;y++) if ( pic1.p[y][x].dd) break;    // find set pixel
    for (h=y;y<pic1.ys;y++) if (!pic1.p[y][x].dd) break;    // find unset pixel
    if (y-h<tr_d) for (;h<y;h++) pic1.p[h][x].dd=0;         // if too small size recolor to zero
    }
pic2.bmp->Canvas->Draw(xx,0,pic1.bmp); xx+=pic1.xs;

See how to extract the borders of an image (OCT/retinal scan image) for the description of picture and color. Or look at any of my DIP/CV tagged answers. Now the code is well commented and straightforward but just need to add:

You can ignore pic2 stuff it is just the image posted above so I do not need to manually print screen and merge the subresult in paint... To improve robustness you should add enhancing of dynamic range (so the tresholds have the same conditions for any input images). Also you should compare to more then just single color (if more wood types of pallet is present).

Now you should segmentate or label the areas

1. loop through entire image
2. find first pixel set with the pallet color

3. flood fill the area with some distinct ID color different from set pallet color

   I use black 0x00000000 space and white 0x00FFFFFF as pallete pixel color. So use ID={1,2,3,4,5...}. Also remember number of filled pixels (that is your area) so you do not need to compute it again. You can also compute bounding box directly while filling.

4. compute and compare features

   You need to experiment with more then one image. To find out what properties are good for detection. I would go for circumference length vs area ratio. and or bounding box size... The circumference can be extracted by simply selecting all pixels with proper ID color neighboring black pixel.

See also similar Fracture detection in hand using image proccessing

Good luck and have fun ...