0

Also trying to get access to color data bytes from color cam of Tango, I was stuck on java API by being able to connect tango Cam to a surface for display (but just OK for display in fact, no easy access to raw data, nor time stamp)... so finally I switch using C API on native code (latest FERMAT lib and header) and follow recommendation I found on stack Overflow by registering a derivated sample code to connectOnFrameAvailable()... (I start using PointCloudActivity sample for that test).

  • First problem I found is somewhat a side effect of registering to that callback, that works usually fine (callbacks gets fire regularly), but then another callback that I also registered, to get xyz clouds, start to fail to fire. Like in sample code I mentioned, clouds are get through a onXYZijAvailable() callback, that the app registers using TangoService_connectOnXYZijAvailable(onXYZijAvailable).

So failing to get xyz callback fired is not happening always, but usually half of the time, during tests, with a awful workaround that is by taking the app in background then foreground again ... this is curious, is this "recover" related to On-pause/On-resume low level stuff??). If someone has clues .... By the way in Java API, same side effect was observed, once connecting cam texture for display (through Tango adequate API ...)

But here is my second "problem", back to acquiring YV12 color data from camera : through registering to TangoService_connectOnFrameAvailable( TangoCameraId::TANGO_CAMERA_COLOR, nullptr, onFrameAvailable) and providing static funtion onFrameAvailable defined like this :

static void onFrameAvailable(void* ctx, TangoCameraId id, const TangoImageBuffer* buffer)
{
   ...
   LOGI("OnFrameAvailable(): Cam frame data received");
   // Check if data format of expected type : YV12 , i.e.
   // TangoImageFormatType::TANGO_HAL_PIXEL_FORMAT_YV12 
   //  i.e.  = 0x32315659  // YCrCb 4:2:0 Planar
   //LOGI("OnFrameAvailable(): Frame data format (%x)", buffer->format);
   .... 
}

the problem is that width, height, stride information of received TangoImageBuffer structure seems valid (1280x720, ...), BUT the format returned is changing every-time, and not the expected magic number (here 0x32315659) ... I am doing something wrong there ? (but other info are OK ...)

Also, there is apparently only one data format defined (YV12 ) here, but seeing Fish Eye images from demo app, it seems grey level image, is it using same (color) format as low level capture than the RGB cam ???

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
Vincent A.
  • 56
  • 5

1 Answers1

0

1) Regarding the image from the camera, I came to the same conclusion you did - only availability of image data is through the C API

2) Regarding the image - I haven't had any issues with YUV, and my last encounter with this stuff was when I wrote JPEG stuff - the format is naked, i.e. it's an organizational structure and has no header information save the undefined metadata in the first line of pixels mentioned here - Here's a link to some code that may help you decode the image in a response to another message here

3) Regarding point cloud returns - Please note this information is anecdotal, and to some degree the product of superstition - what works for me only does that sometimes, and may not work at all for you

  • Tango does seem to have a remarkable knack to simply stop producing point clouds. I think a lot of it has to do with very sensitive timing internally (I wonder if anyone mentioned that Linux ain't an RTOS when this was first crafted)
  • Almost all issues I encounter can be attributed to screwing up the timing where

  • A. Debugging the C level can may point clouds stop coming

  • B. Bugs in the native or java code that cause hiccups in the threads that are handling the callbacks can cause point clouds to stop coming

  • C. Excessive load can cause the system to loose sync, at which point the point clouds will stop coming - this is detectable, you will start to see a silvery grid pattern appear in rectangular areas of the image, and point clouds will cease. Rarely, the system will recover if load decreases, the silvery pattern goes away, and point clouds come back - more commonly the silvery pattern (I think its the 3d spatializing grid) grows to cover more of the image - at least a restart of the app is required for me, and a full tablet reboot every 3rd time or so

Summarizing, that's my suspicions and countermeasures, but it's based completely on personal experience -

Community
  • 1
  • 1
Mark Mullin
  • 1,340
  • 1
  • 9
  • 21
  • I can add I'm now pretty certain that tango is quietly judgmental about how long your callback functions take to execute - if I decode the camera image inside the image callback, then all of a sudden I don't get point cloud callbacks - if I do that later on in the render pass its happy - what's weird is that I think it's the same thread – Mark Mullin Jan 28 '15 at 22:37
  • I did notice that callbacks to get the color image and callbacks to get the fisheye image are running in different threads. And yes, I think you don't want your callback functions to do a lot of processing. I'm wondering if this logcat message has anything do with it: "routeResultFromCAM: WARNING: only 3 requests in flight on sensor 0, frame drops likely". – t2k32316 Jan 29 '15 at 22:05
  • I was actually seeing a lot of that yesterday - somehow it was related to the fact that tango callbacks absolutely should not call back into java, not even just to set the global islocalized flag - I would see that message postmortem after the thing rolled over and died soon after (but not at) the abovementioned crime against tango was committed. – Mark Mullin Jan 29 '15 at 22:59
  • Thanks for your comments. I finally use the render (different?) thread with adequate mutexes to transform yv12 data into RGBA image, but as you all mentionned callbacks timing (and overall system load) seems crucial to get something stable. – Vincent A. Jan 30 '15 at 09:14
  • @VincentAlleaume were you able to get images without artifacts? My images still get weird lines at the bottom of the frame (see 2nd picture [here](http://stackoverflow.com/questions/28157148/save-frame-from-tangoservice-connectonframeavailable)). I'm actually using a program called vooya to display the raw YV12 frame and haven't tried converting to RGBA yet... – t2k32316 Feb 01 '15 at 03:14
  • I also get weird line at the bottom of image, I wonder if it has something related with the first line of original yv12 image data being used for extra information (line that we ignore when reading yv12 data however, so it should be not related). In my case, I will not try to use that last line color info, not a problem at this step) – Vincent A. Feb 02 '15 at 07:03