sketchy timestamp for color image tango jni

Question

I used the tango c api to get color images, point clouds and their timestamps from the google tango. I store the images, point clouds and timestamps in a queue to be processed later since processing during the interrupt routine will cause frames to be dropped. Unfortunately, the time values I get are weird. I don't get point clouds at 5fps and I don't trust the timestamps from the images. Has anyone experienced anything like this?

My code is below. This is the code that attaches my functions to the interrupt routines

TangoErrorType TangoHandler::ConnectCameraCallback() {
// TangoCoordinateFramePair is used to tell Tango Service about the frame of
// references that the applicaion would like to listen to.

TangoCameraId id = TANGO_CAMERA_FISHEYE;


int ret;
//for color images

TangoCameraId id2 = TANGO_CAMERA_COLOR;
ret = TangoService_connectOnFrameAvailable(id2,nullptr, onColorFrameAvailable);
if (ret != TANGO_SUCCESS) {
    LOGE(
            "JNI_App: Failed to connect to camera callback with error"
                    "code: %d",
            ret);
    return TANGO_ERROR;
}

//for point clouds
ret = TangoService_connectOnXYZijAvailable(onPointCloudAvailableCallback);
       if (ret != TANGO_SUCCESS) {
         LOGE(
             "JNI_App: Failed to connect to point cloud callback with error"
             "code: %d",
             ret);
         return TANGO_ERROR;
       }

return TANGO_SUCCESS;


}

This is the code that grabs color frames and puts them in a queue

static void onColorFrameAvailable(void*, TangoCameraId id, const TangoImageBuffer *buffer) {


if(run_producer_thread != 0){


    double timestamp = buffer->timestamp;
    uint32_t height = buffer->height;
    uint32_t width = buffer->width;
    uint32_t stride = buffer->stride;
    int64_t frame_n = buffer->frame_number;


    cv::Mat m = cv::Mat(height + height / 2, width, CV_8UC1, 0, stride);

    cv::Mat m2;

    m.data = buffer->data;
    cv::cvtColor(m, m, CV_YUV2BGR_NV21,3);
    m.copyTo(m2);

    queue_mutex.lock();
    mColorImageQueue.operate("push",NULL,&m2,NULL,NULL);
    queue_mutex.unlock();

    queue_mutex.lock();
    mColorTimestampQueue.operate("push",NULL,&timestamp,NULL,NULL);
    queue_mutex.unlock();


    delete [] m.data;
    m.data = NULL;

    delete [] m2.data;
    m2.data = NULL;

    color_frame_number++;
    LOGI("Stored color frame number: %d ",color_frame_number);
 }

}

I do something similar for the point clouds. I have attached the output of my code. Notice how some of the timestamps are identical (which should not be). Please forgive the style of the code, was just trying to get stuff to work.

For the data below the following explanation might make things a bit easier.

Sensor type is either 4 or 5. 4 means depth sensor, implying a pointcloud reading at the corresponding row while 5 implies and color image reading at the corresponding row.

Time difference is the time elapsed between the next reading and the current one

<html>

<head>
<meta http-equiv=Content-Type content="text/html; charset=macintosh">
<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 14">
<link rel=File-List href="data_description_stack_overflow_files/filelist.xml">
<style>
<!--table
 {mso-displayed-decimal-separator:"\.";
 mso-displayed-thousand-separator:"\,";}
@page
 {margin:1.0in .75in 1.0in .75in;
 mso-header-margin:.5in;
 mso-footer-margin:.5in;}
.style0
 {mso-number-format:General;
 text-align:general;
 vertical-align:bottom;
 white-space:nowrap;
 mso-rotate:0;
 mso-background-source:auto;
 mso-pattern:auto;
 color:black;
 font-size:12.0pt;
 font-weight:400;
 font-style:normal;
 text-decoration:none;
 font-family:Calibri, sans-serif;
 mso-font-charset:0;
 border:none;
 mso-protection:locked visible;
 mso-style-name:Normal;
 mso-style-id:0;}
td
 {mso-style-parent:style0;
 padding-top:1px;
 padding-right:1px;
 padding-left:1px;
 mso-ignore:padding;
 color:black;
 font-size:12.0pt;
 font-weight:400;
 font-style:normal;
 text-decoration:none;
 font-family:Calibri, sans-serif;
 mso-font-charset:0;
 mso-number-format:General;
 text-align:general;
 vertical-align:bottom;
 border:none;
 mso-background-source:auto;
 mso-pattern:auto;
 mso-protection:locked visible;
 white-space:nowrap;
 mso-rotate:0;}
.xl65
 {mso-style-parent:style0;
 color:black;}
-->
</style>
</head>

<body link=blue vlink=purple>

<table border=0 cellpadding=0 cellspacing=0 width=465 style='border-collapse:
 collapse;table-layout:fixed;width:465pt'>
 <col width=65 span=4 style='width:65pt'>
 <col width=101 style='mso-width-source:userset;mso-width-alt:4309;width:101pt'>
 <col width=104 style='mso-width-source:userset;mso-width-alt:4437;width:104pt'>
 <tr height=15 style='height:15.0pt'>
  <td height=15 class=xl65 width=65 style='height:15.0pt;width:65pt'>Time(s)</td>
  <td class=xl65 width=65 style='width:65pt'>Place holder</td>
  <td class=xl65 width=65 style='width:65pt'>Place holder</td>
  <td class=xl65 width=65 style='width:65pt'>Place holder</td>
  <td class=xl65 width=101 style='width:101pt'>sensor type</td>
  <td class=xl65 width=104 style='width:104pt'>time difference(s)</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.305469</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.043884</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.349353</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>4</td>
  <td align=right>0.02165</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.371003</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.40377</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.436537</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.469304</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.502071</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.502071</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.098301</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.600372</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.633139</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.665906</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.665906</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.665906</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.665906</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.065534</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.73144</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.065534</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.796974</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.829741</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.036934</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.866675</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.899442</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.0286</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.928042</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.928042</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.065534</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>108.993576</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.065534</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.05911</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.091877</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.091877</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.124644</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.065534</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.190178</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.222945</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.255712</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.288479</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.321246</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.321246</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.065534</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.38678</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.419547</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.452314</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.485081</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.517848</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.517848</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.065534</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.583382</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.583382</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.583382</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.616149</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.616149</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.616149</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.616149</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.616149</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.616149</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.648916</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.681683</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.032767</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.71445</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.71445</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.71445</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.71445</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.71445</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0.043884</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.758334</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>4</td>
  <td align=right>0.02165</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.779984</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td align=right>0</td>
 </tr>
 <tr height=15 style='height:15.0pt'>
  <td height=15 align=right style='height:15.0pt'>109.779984</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>0</td>
  <td align=right>5</td>
  <td></td>
 </tr>
 <tr height=4920 style='height:4920.0pt;mso-xlrowspan:328'>
  <td height=4920 colspan=6 style='height:4920.0pt;mso-ignore:colspan'></td>
 </tr>
 <tr height=12 style='mso-height-source:userset;height:12.0pt'>
  <td height=12 colspan=6 style='height:12.0pt;mso-ignore:colspan'></td>
 </tr>
</table>

</body>

</html>

I'd recommend commenting out everything except the code that logs the timestamp and see what that looks like. For one thing, it is possible that your data copying and conversion is using up too much time. For another thing, your conversion code might be corrupting memory - I'm not very familiar with the OpenCV API, but that `cvtColor` using the same source and destination looks suspect to me. — rhashimoto, Jan 12 '16 at 01:10
Agree with @rhashimoto, if you block a single callback thread, it blocks all other callbacks as they all coming in same thread. — xuguo, Jan 12 '16 at 23:25
I agree with you guys. That was definitely part of the issue but the biggest culprit was this problem: [link](http://stackoverflow.com/questions/29236110/timing-issues-with-tango-image-frames) — TangoExplorer, Feb 04 '16 at 06:16

sketchy timestamp for color image tango jni

0 Answers0