Android Camera2: How to implement an semi-automatic Shutter Speed Priority Mode

Question

Goal

• Capture images with Android smartphones attached to moving vehicles
• frequency: 1 Hz
• reference model: Google Pixel 3a
• objects of interest: the road/way in front of the vehicle
• picture usage: as input for machine learning (like RNN) to identify damages on the road/way surfaces
• capture environment: outdoor, only on cloudy days

Current state

• Capturing works (currently using JPEG instead of RAW because of the data size)
• auto-exposure works
• static focus distance works

Challenge

• The surface of the ways/roads in the pictures are often blurry
• The source of the motion blur is mostly from the shaking vehicle/fixed phone
• To reduce the motion blur we want to use a "Shutter Speed Priority Mode"
  • i.e. minimize shutter speed => increase ISO (accept increase noise)
  • there is only one aperture (f/1.8) available
  • there is no "Shutter Speed Priority Mode" (short: Tv/S-Mode) available in the Camera2 API
  • the CameraX API does not (yet) offer what we need (static focus, Tv/S Mode)

Steps

• Set the shutter speed to the fastest exposure supported (easy)
• Automatically adjust ISO setting for auto-exposure (e.g. this formular)
• To calculate the ISO the only missing part is the light level (EV)

Question

• How can I estimate the EV continuously during capturing to adjust the ISO automatically while using a fixed shutter speed?

Ideas so far:

1. If I could read out the "recommendations" from the Camera2 auto exposure (AE) routine without actually enabling AE_MODE_ON then I could easily calculate the EV. However, I did not find an API for this so far. I guess it's not possible without routing the device.
2. If the ambient light sensor would provide all information needed to auto-expose (calculate EV) this would also be very easy. However, from my understanding it only measures the incident light not the reflected light so the measurement does not take the actual objects in the pictures into account (how their surfaces reflect light)
3. If I could get the information from the Pixels of the last captures this would also be doable (if the calculation time fits into the time between two captures). However, from my unterstanding the pixel "bightness" is heavily dependent on the objects captured, i.e. if the brightness of the objects captured changes (many "black horses" or "white bears" at the side of the road/way) I would calculate bad EV values.
4. Capture auto-exposed images in-between the actual captures and calculate the light levels from the auto-selected settings used in the in-between captures for the actual captures. This would be a relatively "good" way from my understanding but it's quite hard on the resources end - I am not sure I the time available between two captures is enough for this.

Maybe I don't see a simpler solution. Has anyone done something like this?

Answer 1

Yes, you need to implement your own auto-exposure algorithm. All the 'real' AE has to go by is the image captured by the sensor as well, so in theory you can build something just as good at guessing the right light level.

In practice, it's unlikely you can match it, both because you have a longer feedback loop (the AE algorithm can cheat a bit on synchronization requirements and update sensor settings faster than an application can), and because the AE algorithm can use hardware statistics units (collect histograms and average values across the scene), which make it more efficient.

But a simple auto-exposure algorithm would be to average the whole scene (or a section of the scene, or every-tenth-pixel of the scene, etc) and if that average is below half max value, increase ISO, and if it's above, reduce. A basic feedback control loop, in other words. With all the issues about stability, convergence, etc, that apply. So a bit of control theory understanding can be quite helpful here. I'd recommend a low-resolution YUV output (640x480 maybe?) to an ImageReader from the camera to use as the source data, and then just look at the Y channel. Not a ton of data to churn through in that case.

Or as hb0 mentioned, if you have a very limited set of outdoor conditions, you can try to hardcode values for each of them. But the range of outdoor brightness can be quite large, so this would require a decent bit of testing to make sure it'll work, plus manual selection of the right values each time.

Answer 2

When the pictures are only captured in specific light situations like "outdoor, cloudy":

Tabulated values can be used for the exposure value (EV) instead of using light measurements.

Example

• EV100 (iso100) for Outdoor cloudy (OC) = 13
• EV (dynamic iso) for OC = EV100 + log2(iso/100)

Using this formula together with those formulas we can calculate the iso from:

• aperture (fixed)
• shutter speed (manually selected)

Additionally, we could add an UI option to choose a "light situation" like:

• outdoor, cloudy
• outdoor, sunny
• etc.

This is probably not the most accurate way but for now a first, easy way to continue prototyping.