12

We would like to process 120 fps (or 60fps) camera preview data using Neon assembly or GPU shaders to perform a simple image processing operation (eg 3x3 convolution). We are NOT interested in displaying the image on the screen so the camera data should go directly to a SurfaceTexture.

We thought we could do this using Camera2 API on a Nexus 5 using CameraConstrainedHighSpeedCaptureSession but have been unsuccessful.

There is no currently functional code on the internet (at least the part we have scraped) that allows this and yet it seems probable that the functionality exists.

The best example of using the Camera2 API for highspeed recording is here: https://github.com/PkmX/lcamera

In order to try to encourage someone to post a solution I am putting a bounty on this question.

twerdster
  • 4,977
  • 3
  • 40
  • 70

2 Answers2

2

This is accomplished in pjsip. Pull the repo to your machine:

svn checkout -r 4687 https://svn.pjsip.org/repos/pjproject/trunk

Then open the pjmedia/src/swig folder for an example app that renders video data incoming from the wire directly to a SurfaceTexture.

I have uploaded the two most relevant C source files (at a glance—there are certainly others) for you, here:

  1. android_dev.c
  2. android_opengl.c

Naturally, those files contain lots of irrelevant code for you, but hopefully have the snippet you're looking for.

You may need to first follow some of the instructions found here in order to generate the pjmedia/src/pjmedia-videodev/android/PjCamera*.java files that handle the minimalist Java/C interfacing if you are unsure how to do this. I was able to follow their instructions—using exactly the same versions for the NDK (works w/10d or 10e) and openssl (1.0.2a)—and successfully render video and create a custom app. This source accomplishes exactly what you're looking for if you just modify the C-level video capture to use your API via JNI hooks the same way they do.

For an example of using JNI hooks to change the API, have a look at their audiodev folder in pjmedia.

If you do build their project, be sure to follow the instructions at the very bottom for the newer NDK versions for building openssl 1.0.2a. Note the capital 'C' on ./Configure, as executing the standard ./configure may launch the wrong configuration process. I suggest copy and pasting their provided commands into a script. You may be able to skip openssl entirely for your purposes; I was only able to get my Android devices registering SIP accounts consistently once I integrated the openssl support.

EntangledLoops
  • 1,951
  • 1
  • 23
  • 36
  • Thanks for the detailed reply. Have you managed to get 60 or 120 fps out of the stream ? – twerdster May 13 '16 at 17:02
  • It depends on the device, but definitely: between 2 modern devices, I can stream from the camera in realtime at max resolution, which is (for me) capped usually at either 30 or 60 fps (w/communication of the full stream to a second device for rendering as in pjsip). On a local device rendering directly to a hidden texture, the maximum framerate I've seen is ~200 fps. – EntangledLoops May 13 '16 at 17:04
  • 60fps from the camera? What do you mean by 200 fps? Im specifically asking about frames coming from the physical device camera. And which devices are you using? – twerdster May 13 '16 at 17:06
  • Sorry, I meant OpenGL was capable of hitting high framerates while simultaneously capturing/copying the video data (at 30 or 60 fps) to a hidden surface. The camera is totally dependent upon the device. I personally haven't captured at 120 fps, but I also haven't tried (speed isn't a factor yet for my video project). However, the JNI hooks are slow that I'm using and it still keeps up fine. – EntangledLoops May 13 '16 at 17:08
  • Okay, so if I understand you correectly you are able to capture at 60 fps and process much faster on full resolution. What resolution and Which device are you using? – twerdster May 13 '16 at 17:10
  • Yes. I'm using a 1280x720 resolution and an ASUS Zenfone Z00TD presently. Also should mention to checkout the pjsip instructions for native codec support on Android. I'm focused on audio so my video is just minimal-working example for the time being, but I'm pleased w/the performance for now. – EntangledLoops May 13 '16 at 17:11
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/111882/discussion-between-entangledloops-and-twerdster). – EntangledLoops May 13 '16 at 17:15
2

We succesfully modified the camera2video sample to obtain 120 fps data rate on a Samsung S6.

The code is here.

We have not yet managed to extract the pixels but will update here when we do.

twerdster
  • 4,977
  • 3
  • 40
  • 70
  • I saw the code you've shared above. And have two things to say about it: 1. It is not so stable and has some issues when it is to work for 120 or above fps recording (high speed recording) 2. The code is not meant to achieve 60 fps recording at all. Which I am right now looking for. So, do you have anything that helps to device if the device supports 60fps recording? – Ashutosh Tiwari Jan 10 '20 at 13:13