Extract colors at known coordinates from reference patterns

Question

I have a set of colors in a card, as below (I can modify the card as required)

I have 4 reference points at four corners. I have a bunch of colors at known coordinates (in mm) from these reference points.

A user clicks a picture of this card on their phone (Android) (assume good lighting condition)

I can calibrate the distance from the number of pixels between the identified reference patterns and the known actual distance (in mm) between them.

Pixels per mm = (Number of pixels between the centers of any two reference points) / (Actual distances in mm between those two reference points)

How do I extract each of these color values from the picture on the Android phone locally?

What I thought so far:

1. Detect the reference patterns and get their coordinates.

2. I know the actual distance in mm

3. From the coordinates and actual distance, get pixels-per-mm

4. I know the actual distance in mm of each of these colors from the reference points

5. Offset by that distance for each of the colors and find the average color in a small radius around that point.

How do I achieve this through code? And particularly on Android.