10

I am trying to modify the source for screenrecord in android 4.4 and lower the captured frame rate, but no matter what value I put in: 

format->setFloat("frame-rate", 5);

the result is always the same ( a very high frame rate )

Is the encoder ignoring this property ? how can I control the frame rate ?

jacob
  • 1,397
  • 1
  • 26
  • 53
  • 2
    It looks like that value is only used for writing some header info into the resulting mp4 file, not for controlling the capture rate. For that, you'd have to modify the encoder loop and drop frames accordingly. – marcone Mar 11 '14 at 21:33

1 Answers1

11

The frame-rate value is not ignored, but it doesn't do what you want.

The combination of frame-rate and i-frame-interval determines how often I-frames (also called "sync frames") appear in the encoded output. The frame rate value might also play a role in meeting the bitrate target on some devices, but I'm not sure about that (see e.g. this post).

The MediaCodec encoder does not drop frames. If you want to reduce the frame rate, you have to do so by sending fewer frames to it.

The screenrecord command doesn't "sample" the screen at a fixed frame rate. Instead, every frame it receives from the surface compositor (SurfaceFlinger) is sent to the encoder, with an appropriate time stamp. If screenrecord receives 60 frames per seconds, you'll have 60fps output. If it receives 10 frames in quick succession, followed by nothing for 5 seconds, followed by a couple more, you'll have exactly that in the output file.

You can modify screenrecord to drop frames, but you have to be a bit careful. If you try to reduce the maximum frame rate from 60fps to 30fps by dropping every-other frame, you run the risk that in a "frame0 - frame1 - long_pause - frame2" sequence you'll drop frame1, and the video will hold on frame0 instead, showing a not-quite-complete animation. So you need to buffer up a frame, and then encode or drop frame N-1 if the difference in presentation times between that and frame N is ~17ms.

The tricky part is that screenrecord, in its default operating mode, directs the frames to the encoder without touching them, so all you see is the encoded output. You can't arbitrarily drop individual frames of encoded data, so you really want to prevent the encoder from seeing them in the first place. If you use the screenrecord v1.1 sources you can tap into "overlay" mode, used for --bugreport, to have the frames pass through screenrecord on their way to the encoder.

In some respects it might be simpler to write a post-processor that reduces the frame rate. I don't know how much quality would be lost by decoding and re-encoding the video.

Update: for an example of how to do it crudely, add this to processFrame_l():

     int64_t droppedFrames = 0;
+    {
+        static int flipflop = 0;
+        flipflop = 1 - flipflop;
+        if (flipflop) {
+            printf("dropping frame %lld\n", frameNumber);
+            return;
+        }
+    }

     if (mLastFrameNumber > 0) {

Note this comes after updateTexImage(), which acquires the next buffer, and skips the call to swapBuffers(), which submits the buffer to the video encoder.

Community
  • 1
  • 1
fadden
  • 51,356
  • 5
  • 116
  • 166
  • I am trying to use screenrecord v 1.1 to set frame rate but with no luck, if I just skip frames in threadLoop it doesnt work. how can I successfully skip frames ? – jacob Mar 26 '14 at 08:18
  • and if I add sleep in the threadLoop it slows down the device – jacob Mar 26 '14 at 09:58
  • your patch works great, but it has a problem with the last frame. when no frames are arriving. the last frame isnt captured. can I force the last frame to repeat if no frames arrive ? – jacob Mar 26 '14 at 18:26
  • This is the problem I talked about in my answer ("you have to be a bit careful"). I don't know a simple way to do it right. You can call `updateTexImage()` when nothing new is available and it'll just get the previous frame again, so you may just need to have a timer go off when no new frames have arrived, and submit the previous frame if it hadn't been. – fadden Mar 26 '14 at 18:30
  • the timer idea sounds like it can work, can you help me with some example code? btw would KEY_REPEAT_PREVIOUS_FRAME_AFTER work here ? – jacob Mar 26 '14 at 18:40
  • can I tell the producer to push the screen again ? – jacob Mar 26 '14 at 19:13
  • The encoder will see whatever was drawn before `swapBuffers()`. The code in `processFrame_l()` latches the new frame, renders it, and swaps. You could split that, changing the main loop to call `updateTexImage()` whenever a new frame is available, but not draw anything; then every 33.3ms you wake up and do the render + swap. This will yield video at a constant 30fps, submitting the same frame repeatedly when the system is idle. If you don't want that, you can add a "have I submitted this frame" test that resets when a new frame arrives. – fadden Mar 26 '14 at 19:23
  • a constant feed every 33ms sounds good, but I dont fully understand what you suggested. if no new frames are arriving and I call swapBuffer it will remain with the last frame? what exactly does swapbuffer do ? – jacob Mar 26 '14 at 20:05
  • 1
    Think of `swapBuffers()` as "submit whatever I just drew to the video encoder". If you just call `swapBuffers()` without drawing anything you will be unhappy. `updateTexImage()` means "grab whatever the most recent thing we got from the compositor is, and put it in a GLES texture". So if you look at `processFrame_l()`, you can see that it grabs the new frame (or the old frame if nothing new appeared), renders it with some text, and then submits it to the video encoder. – fadden Mar 26 '14 at 21:34
  • as usual you suggestion works great. I added a thread that wakes the overly according to required frame rate. this way I dont loose the last frame. but I thought that the encoder would enhance the output if it would get the same frame over and over ( when using low bitrate ) , but it doesn't. any way to tell it to do that ? – jacob Mar 27 '14 at 00:24
  • I'm not sure what you mean by "enhance the output". – fadden Mar 27 '14 at 03:30
  • by enhanced I mean less pixelated ( as in x264 that when the changes between frames are small the encoder overtime generates a less pixelated output ) – jacob Mar 27 '14 at 08:35
  • That's up to the H.264 encoder. I don't know what behavior you can rely on there. – fadden Mar 27 '14 at 14:41
  • ok, do you know how I can disable the run-time rotation change , that is I want the captured frames to stay the same (the native orientation) even if the device is rotated – jacob Mar 27 '14 at 15:32
  • I don't know that there's a way to do that with a virtual display -- everything goes through the display projection set by `setDisplayProjection()`. You can disable automatic rotation for the device in Settings, but I don't know if that's an option for you. – fadden Mar 27 '14 at 19:31
  • maybe i can rotate the frames in the overlay? would it increase processing time allot? how can i do that? – jacob Mar 27 '14 at 23:26
  • The orientation change is discovered by screenrecord through polling (which is why it doesn't work quite right). You can see the setDisplayProjection() call in runEncoder(). It should be a "simple" matter of passing the right set of args to SurfaceComposerClient::setDisplayProjection(). This is an undocumented private API, so the only info about this method is in the header: https://android.googlesource.com/platform/frameworks/native/+/kitkat-release/include/gui/SurfaceComposerClient.h – fadden Mar 28 '14 at 00:01
  • but wont setDisplayProjection switch my capture Height and Width ? the problem is that I am streaming the encoded output via rtp and the player doesn't know about the dimensions change. wont rotating the frames in the overlay solve this problem ? – jacob Mar 28 '14 at 08:20
  • Try removing the display projection changes in runEncoder(), and watch what happens. When you rotate the device, the output will be cut off. SurfaceFlinger is doing the composition in the new orientation, so you have to follow. You need to update the projection to cover the new dimensions of the virtual display. Since you already have to provide the display projection, you might as well rotate it there too. – fadden Mar 28 '14 at 14:43