On Android, I want to extract all frames of a video and run an object tracking function from boofcv on each frame. Therefore, I am using the ExtractMpegFramesTest example and slightly adjusted it to run the tracking on each frame instead of saving the frames as png. I.e. instead of calling outputSurface.saveFrame() I am calling outputSurface.processFrame(), which I implemented as follows:
public void processFrame() {
mPixelBuf.rewind();
GLES20.glReadPixels(0, 0, mWidth, mHeight, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE,
mPixelBuf);
Bitmap bmp = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
mPixelBuf.rewind();
bmp.copyPixelsFromBuffer(mPixelBuf);
GrayU8 img = ConvertBitmap.bitmapToGray(bmp, (GrayU8)null, null);
Log.d(TAG, "Running tracker...");
tracker.process(img, location);
barPath.add(getCenterOf(location));
Log.d(TAG, "Finished running tracker.");
bmp.recycle();
}
As soon as I comment out the tracker.process(img, location) line, the code works flawlessly. As soon as I include the tracking, I receive no error meassage but also nothing happens. The app gets stuck at tracking the first image, the logcat output looks like this:
2021-10-19 13:01:05.778 32449-32449/org.boofcv.android D/ExtractMpegFrames: Initialized
2021-10-19 13:01:05.780 32449-32449/org.boofcv.android I/Choreographer: Skipped 30 frames! The application may be doing too much work on its main thread.
2021-10-19 13:01:05.789 32449-32449/org.boofcv.android W/Looper: Slow Looper main: doFrame is 514ms late because of 3 msg, msg 2 took 510ms (seq=242 running=153ms runnable=2ms late=18ms h=android.view.ViewRootImpl$ViewRootHandler c=android.view.View$PerformClick)
2021-10-19 13:01:05.793 32449-32726/org.boofcv.android D/ExtractMpegFrames: Extractor selected track 0 (video/avc): {track-id=1, file-format=video/mp4, level=8192, mime=video/avc, frame-count=624, profile=8, language=, color-standard=1, display-width=1920, csd-1=java.nio.HeapByteBuffer[pos=0 lim=9 cap=9], color-transfer=3, durationUs=10395844, display-height=1080, width=1920, color-range=2, rotation-degrees=90, max-input-size=1555201, frame-rate=60, height=1080, csd-0=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]}
2021-10-19 13:01:05.794 32449-32726/org.boofcv.android D/ExtractMpegFrames: Video size is 1920x1080
2021-10-19 13:01:05.824 32449-32726/org.boofcv.android D/ExtractMpegFrames: textureID=1
2021-10-19 13:01:05.830 32449-32728/org.boofcv.android I/OMXClient: IOmx service obtained
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: connecting to surface 0x7864f16010, reason connectToSurface
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android I/MediaCodec: [OMX.qcom.video.decoder.avc] setting surface generation to 33227777
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: disconnecting from surface 0x7864f16010, reason connectToSurface(reconnect)
2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: connecting to surface 0x7864f16010, reason connectToSurface(reconnect)
2021-10-19 13:01:05.890 32449-32728/org.boofcv.android I/ExtendedACodec: setupVideoDecoder()
2021-10-19 13:01:05.892 32449-32728/org.boofcv.android I/ExtendedACodec: Decoder will be in frame by frame mode
2021-10-19 13:01:05.929 32449-32728/org.boofcv.android D/SurfaceUtils: set up nativeWindow 0x7864f16010 for 1920x1080, color 0x7fa30c06, rotation 90, usage 0x20002900
2021-10-19 13:01:05.939 32449-32728/org.boofcv.android W/Gralloc3: allocator 3.x is not supported
2021-10-19 13:01:05.939 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.946 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 0 to dec, size=148096
2021-10-19 13:01:05.958 32449-32726/org.boofcv.android D/ExtractMpegFrames: no output from decoder available
2021-10-19 13:01:05.958 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.959 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 1 to dec, size=14112
2021-10-19 13:01:05.970 32449-32726/org.boofcv.android D/ExtractMpegFrames: no output from decoder available
2021-10-19 13:01:05.970 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.972 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 2 to dec, size=17632
2021-10-19 13:01:05.974 32449-32726/org.boofcv.android D/ExtractMpegFrames: decoder output format changed: {crop-right=1919, color-format=2141391878, slice-height=1088, mime=video/raw, hdr-static-info=java.nio.HeapByteBuffer[pos=0 lim=25 cap=25], stride=1920, color-standard=2, color-transfer=3, crop-bottom=1079, crop-left=0, width=1920, color-range=1, crop-top=0, height=1088}
2021-10-19 13:01:05.974 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
2021-10-19 13:01:05.975 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 3 to dec, size=30208
2021-10-19 13:01:05.976 32449-32726/org.boofcv.android D/ExtractMpegFrames: surface decoder given buffer 16 (size=8)
2021-10-19 13:01:05.976 32449-32726/org.boofcv.android D/ExtractMpegFrames: awaiting decode of frame 0
2021-10-19 13:01:05.978 32449-32449/org.boofcv.android D/ExtractMpegFrames: new frame available
2021-10-19 13:01:05.992 32449-32726/org.boofcv.android D/ExtractMpegFrames: Running tracker...
2021-10-19 13:01:06.003 32449-32727/org.boofcv.android D/SurfaceUtils: disconnecting from surface 0x7864f16010, reason disconnectFromSurface
The tracker.process() functions itself works flawlessly, e.g. when using MediaMetadataRetriever to extract the frames (however MediaMetadataRetriever is too slow, that's why I am using ExtractMpegFrames). On my device the tracker.process() function itself usually takes about 30-40 ms per frame. Just extracting the frames with ExtractMpegFrames as bitmap without further processing takes about 3-4 ms per frame.
So I guess the problem might have to do with threading? I would be thankful for every help.
Edit
On my UI Thread I am calling new ExtractMpegFrames().run() while the ExtractMpegFrames class looks like this:
public class ExtractMpegFrames {
public void run() throws Throwable {
this.initialize();
ExtractMpegFramesWrapper.execute(this);
}
private static class ExtractMpegFramesWrapper implements Runnable {
private Throwable mThrowable;
private ExtractMpegFrames mExtractor;
private ExtractMpegFramesWrapper(ExtractMpegFrames test) {
mExtractor = test;
}
@Override
public void run() {
try {
mExtractor.extractMpegFrames();
} catch (Throwable th) {
mThrowable = th;
}
}
public static void execute(ExtractMpegFrames obj) throws Throwable {
ExtractMpegFramesWrapper wrapper = new ExtractMpegFramesWrapper(obj);
Thread th = new Thread(wrapper, "main");
th.start();
//th.join();
if (wrapper.mThrowable != null) {
throw wrapper.mThrowable;
}
}
}
...
// The rest (except for the processFrame() function above)
// looks the same as in ExtractMpegFramesTest
// See here:
// https://bigflake.com/mediacodec/#ExtractMpegFramesTest
}
Edit 2
I still did not resolve this issue. I tried to replace the boofcv tracker with an opencv one (org.opencv.tracking.TrackerMOSSE). However, this leads to a deadlock as well (same logcat output as above). Lastly, I tried to replace the tracking function by this opencv template matching algorithm. This works and does not lead to a deadlock, however it is way too slow (takes about one second per frame on my device).
