How to elegantly serialize and deserialize OpenCV YAML calibration data in Java?

Question

I'm trying to load / save OpenCV calibration data in YAML format using the official OpenCV Java bindings. I am aware OpenCV (c++ version at least) can serialize to XML and JSON but I would like to support older YAML calibration files.

A calibration file looks like this:

%YAML:1.0
cameraMatrix: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 6.6278599887122368e+02, 0., 3.1244256016006659e+02, 0.,
       6.6129276875199082e+02, 2.2747179767124251e+02, 0., 0., 1. ]
imageSize_width: 640
imageSize_height: 480
sensorSize_width: 0
sensorSize_height: 0
distCoeffs: !!opencv-matrix
   rows: 5
   cols: 1
   dt: d
   data: [ -1.8848338341464690e-01, 1.0721890419183855e+00,
       -3.5244467228016116e-03, -7.0195032848241403e-04,
       -2.0412827999027101e+00 ]
reprojectionError: 2.1723265945911407e-01

I had a look at a few answer already here and here, however I'm looking for an elegant solution as I haven't quite understood to best map java classes to YAML and back. I've tried a few libraries like jyaml, yamlbeans (both 1.0 from SourceForge and 1.13 via Maven Central) and SnakeYAML.

My current attempt at deserialising sort of works but feels quite hacky:

CalibrationParseTest.java

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;

import org.opencv.core.Core;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.constructor.Constructor;

public class CalibrationParseTest {

    public static void main(String[] args) {
        // load OpenCV native
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

        String yamlPath = "./data/calibration.yml";

        try{

          String yamlString = new String(Files.readAllBytes(Paths.get(yamlPath)), StandardCharsets.UTF_8);
          // remove %YAML:1.0 to avoid scan directive error
          yamlString = yamlString.replaceAll("%YAML:1.0", "");
          // map custom class
          yamlString = yamlString.replaceAll("opencv-matrix", "MatYAML");

          System.out.println("<loaded>");
          System.out.println(yamlString);
          System.out.println("</loaded>");

          Yaml yaml = new Yaml(new Constructor(CalibrationData.class));
          CalibrationData data = yaml.load(yamlString);
          // currently manually parsing data from the HashMap: can this be better ?
          data.populateCV();
          // double check data
          System.out.println("<deserialized>");
          System.out.println(data);
          System.out.println("</deserialized>");

        }catch (IOException e) { 
          e.printStackTrace(); 
        } 
    }

}

CalibrationData.java

import java.util.HashMap;

import org.opencv.core.Mat;
import org.opencv.core.Size;

public class CalibrationData extends HashMap{

    public Mat cameraMatrix;
    public Size imageSize;
    public Size sensorSize;
    public Mat distCoeffs;
    public float reprojectionError;

    public CalibrationData(){}

    public void populateCV(){
        cameraMatrix      = ((MatYAML)get("cameraMatrix")).toMat();
        imageSize         = new Size((int)get("imageSize_width"),(int)get("imageSize_height"));
        sensorSize        = new Size((int)get("sensorSize_width"),(int)get("sensorSize_height"));
        distCoeffs        = ((MatYAML)get("distCoeffs")).toMat();
        reprojectionError = (float)((double)get("reprojectionError"));
    }

    public String toString(){
        if(cameraMatrix == null){
            return String.format("[CalibrationData (not parsed to CV-> call populateCV()\n\tdata: %s\n]",super.toString());
        }
        return String.format("[CalibrationData\n" + 
                             "\tcalibrationMatrix: %s\n" + 
                             "\timageSize: %s\n" + 
                             "\tsensorSize: %s\n" + 
                             "\tdistCoeffs: %s\n" + 
                             "\treprojectionError: %f\n]", cameraMatrix.dump(), imageSize.toString(), sensorSize.toString(), distCoeffs.dump(), reprojectionError);
    }

}

MatYAML.java

import java.util.List;

import org.opencv.core.CvType;
import org.opencv.core.Mat;

public class MatYAML{

    public int rows;
    public int cols;
    public String dt;
    public List<Double> data;

    Mat toMat(){
        Mat out = new Mat(rows, cols, dt.equals("d") ? CvType.CV_64F : CvType.CV_32F);

        int index = 0;

        for (int row = 0; row < rows; row++) {
            for (int col = 0; col < cols; col++) {
                out.put(row, col, data.get(index++));
            }
        }

        return out;
    }

}

This outputs the expected result:

<loaded>

cameraMatrix: !!MatYAML
   rows: 3
   cols: 3
   dt: d
   data: [ 6.6278599887122368e+02, 0., 3.1244256016006659e+02, 0.,
       6.6129276875199082e+02, 2.2747179767124251e+02, 0., 0., 1. ]
imageSize_width: 640
imageSize_height: 480
sensorSize_width: 0
sensorSize_height: 0
distCoeffs: !!MatYAML
   rows: 5
   cols: 1
   dt: d
   data: [ -1.8848338341464690e-01, 1.0721890419183855e+00,
       -3.5244467228016116e-03, -7.0195032848241403e-04,
       -2.0412827999027101e+00 ]
reprojectionError: 2.1723265945911407e-01

</loaded>
<deserialized>
[CalibrationData
    calibrationMatrix: [662.7859988712237, 0, 312.4425601600666;
  0, 661.2927687519908, 227.4717976712425;
  0, 0, 1]
    imageSize: 640x480
    sensorSize: 0x0
    distCoeffs: [-0.1884833834146469; 1.072189041918385; -0.003524446722801612; -0.000701950328482414; -2.04128279990271]
    reprojectionError: 0.217233
]
</deserialized>

Is there a more elegant way of serializing/deserializing between Java OpenCV classes and YAML without these hacks ?

By hacks I mean:

• manually removing yaml version directive
• swapping opencv-matrix with MatYAML string
• manually casting HashMap values
• potentially avoiding to manually populate OpenCV Mat data ? (if possible ?)

Update 2 amanin's answer is cleaner and makes it possible to avoid hackily replacing "!!opencv-matrix", however it doesn't serializing/deserializing Mat:

OpenCVConfig{imageSize_width=640, imageSize_height=480, sensorSize_width=0, sensorSize_height=0, camerMatrix=Matrix{rows=3, cols=3, dt=d, data=[662.7859988712237, 0.0, 312.4425601600666, 0.0, 661.2927687519908, 227.4717976712425, 0.0, 0.0, 1.0]}, distCoeffs=Matrix{rows=5, cols=1, dt=d, data=[-0.1884833834146469, 1.0721890419183855, -0.0035244467228016116, -7.01950328482414E-4, -2.04128279990271]}}
---
imageSize_width: 640
imageSize_height: 480
sensorSize_width: 0
sensorSize_height: 0
reprojectionError: 0.21723265945911407
cameraMatrix:
  rows: 3
  cols: 3
  dt: "d"
  data:
 - 662.7859988712237
 - 0.0
 - 312.4425601600666
 - 0.0
 - 661.2927687519908
 - 227.4717976712425
 - 0.0
 - 0.0
 - 1.0
distCoeffs:
  rows: 5
  cols: 1
  dt: "d"
  data:
 - -0.1884833834146469
 - 1.0721890419183855
 - -0.0035244467228016116
 - -7.01950328482414E-4
 - -2.04128279990271

Please advise on integrating a solution with org.opencv.core.Mat

Answer 1

Have you looked at Jackson library ? It allows mapping JSON/Yaml content to Java POJOs.

I've made a little example which solves two of your problems:

• swapping opencv-matrix with MatYAML string
• manually casting HashMap values

However, for yaml version directive, as it looks like it is not valid Yaml, I'm not sure how to handle it. In my example, I've removed it manually before-hand. Surely, a better solution can be found, but I don't know it.

EDIT2: For the matrix object, I've made a dumb POJO, used internally by Jackson to read brut YAML. Then, I've added a conversion layer (see @JsonSerialize and @JsonDeserialize anotations on OpenCVConfig class) to convert this simple POJO to specialized OpenCV matrix. Jackson offers various technics (streaming, custom converters/deserializers, guiding annotations, etc.) of mapping, so you can explore its capabilities to find the solution that fits best to you need.

So, to make the example work, you'll need two dependencies (given here in maven format):

        <!-- (De)serialization engine -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.10.0</version>
        </dependency>

        <!-- Yaml support -->
        <dependency>
            <groupId>com.fasterxml.jackson.dataformat</groupId>
            <artifactId>jackson-dataformat-yaml</artifactId>
            <version>2.10.0</version>
        </dependency>

Here are maven repository description pages:

Jackson-databind: Via maven search or Via mvnrepository

Jackson-dataformat-yaml: Via maven search or Via mvnrepository

Note : My example contains lot of boilerplate, as I manually (well, IDE) generated getters/setters. You should by far reduce code amount by either:

• Using Lombok lib
• Coding with Kotlin (data classes)
• Using public attributes without getter/setter
• Using java 14 record (not sure if it works at this time)

package fr.amanin.stackoverflow;

import java.util.Arrays;
import java.util.stream.Stream;

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.fasterxml.jackson.databind.type.TypeFactory;
import com.fasterxml.jackson.databind.util.Converter;
import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator;
import com.fasterxml.jackson.dataformat.yaml.YAMLMapper;
import org.opencv.core.CvType;
import org.opencv.core.Mat;

public class YAMLOpenCV {
    /**
     * Engine in charge of YAML decoding.
     */
    public static final ObjectMapper OPENCV_YAML_MAPPER = new YAMLMapper();

    public static void main(String[] args) throws Exception {
        nu.pattern.OpenCV.loadShared();
        System.loadLibrary(org.opencv.core.Core.NATIVE_LIBRARY_NAME);
        final String confStr =
                "cameraMatrix: !!opencv-matrix\n" +
                        "   rows: 3\n" +
                        "   cols: 3\n" +
                        "   dt: d\n" +
                        "   data: [ 6.6278599887122368e+02, 0., 3.1244256016006659e+02, 0.,\n" +
                        "       6.6129276875199082e+02, 2.2747179767124251e+02, 0., 0., 1. ]\n" +
                        "imageSize_width: 640\n" +
                        "imageSize_height: 480\n" +
                        "sensorSize_width: 0\n" +
                        "sensorSize_height: 0\n" +
                        "distCoeffs: !!opencv-matrix\n" +
                        "   rows: 5\n" +
                        "   cols: 1\n" +
                        "   dt: d\n" +
                        "   data: [ -1.8848338341464690e-01, 1.0721890419183855e+00,\n" +
                        "       -3.5244467228016116e-03, -7.0195032848241403e-04,\n" +
                        "       -2.0412827999027101e+00 ]\n" +
                        "reprojectionError: 2.1723265945911407e-01";

        OpenCVConfig conf = OPENCV_YAML_MAPPER.readValue(confStr, OpenCVConfig.class);
        System.out.println(conf);

        String serialized = OPENCV_YAML_MAPPER.writeValueAsString(conf);
        System.out.println(serialized);
    }

    /**
     * Java model mirroring YAML configuration. Jackson will fill it 
     * with values read from YAML configuration file, matching YAML 
     * fields with this class property names.
     */
    public static class OpenCVConfig {
        int imageSize_width;
        int imageSize_height;
        int sensorSize_width;
        int sensorSize_height;

        double reprojectionError;

        /* Special case: Matrix objects are decoded in two passes:
         * 1. Jackson will check below converters, and use their input 
         * type to decode YAML to `Matrix` object (intermediate step). 
         * 2. Jackson uses converter to delegate to user the mapping 
         * from this intermediate POJO to specialized target (`Mat` here) 
         */
        @JsonDeserialize(converter = ToMatConverter.class)
        @JsonSerialize(converter = FromMatConverter.class)
        Mat cameraMatrix;
        @JsonDeserialize(converter = ToMatConverter.class)
        @JsonSerialize(converter = FromMatConverter.class)
        Mat distCoeffs;

        public int getImageSize_width() {
            return imageSize_width;
        }

        public OpenCVConfig setImageSize_width(int imageSize_width) {
            this.imageSize_width = imageSize_width;
            return this;
        }

        public int getImageSize_height() {
            return imageSize_height;
        }

        public OpenCVConfig setImageSize_height(int imageSize_height) {
            this.imageSize_height = imageSize_height;
            return this;
        }

        public int getSensorSize_width() {
            return sensorSize_width;
        }

        public OpenCVConfig setSensorSize_width(int sensorSize_width) {
            this.sensorSize_width = sensorSize_width;
            return this;
        }

        public int getSensorSize_height() {
            return sensorSize_height;
        }

        public OpenCVConfig setSensorSize_height(int sensorSize_height) {
            this.sensorSize_height = sensorSize_height;
            return this;
        }

        public double getReprojectionError() {
            return reprojectionError;
        }

        public OpenCVConfig setReprojectionError(double reprojectionError) {
            this.reprojectionError = reprojectionError;
            return this;
        }

        public Mat getCameraMatrix() {
            return cameraMatrix;
        }

        public OpenCVConfig setCameraMatrix(Mat cameraMatrix) {
            this.cameraMatrix = cameraMatrix;
            return this;
        }

        public Mat getDistCoeffs() {
            return distCoeffs;
        }

        public OpenCVConfig setDistCoeffs(Mat distCoeffs) {
            this.distCoeffs = distCoeffs;
            return this;
        }

        @Override
        public String toString() {
            return "OpenCVConfig{" +
                    "imageSize_width=" + imageSize_width +
                    ", imageSize_height=" + imageSize_height +
                    ", sensorSize_width=" + sensorSize_width +
                    ", sensorSize_height=" + sensorSize_height +
                    ", camerMatrix=" + cameraMatrix +
                    ", distCoeffs=" + distCoeffs +
                    '}';
        }
    }

    /**
     * Converter used for serialization of Mat objects into YAML.
     */
    private static class FromMatConverter implements Converter<Mat, Matrix> {

        @Override
        public Matrix convert(Mat value) {
            final Matrix result = new Matrix();
            result.cols = value.cols();
            result.rows = value.rows();
            final int type = value.type();
            result.dt = Stream.of(MatrixDataType.values())
                    .filter(dt -> dt.mapping == type)
                    .findAny()
                    .orElseThrow(() -> new UnsupportedOperationException("No matching datatype found for "+type));
            int idx = 0;
            result.data = new double[result.rows * result.cols];
            for (int r = 0 ; r < result.rows ; r++) {
                for (int c = 0; c < result.cols; c++) {
                    final double[] v = value.get(r, c);
                    result.data[idx++] = v[0];
                }
            }
            return result;
        }

        @Override
        public JavaType getInputType(TypeFactory typeFactory) {
            return typeFactory.constructType(new TypeReference<Mat>() {});
        }

        @Override
        public JavaType getOutputType(TypeFactory typeFactory) {
            return typeFactory.constructType(new TypeReference<Matrix>() {});
        }
    }

    /**
     * Converter used at read time, to map YAML object to OpenCV Mat.
     */
    private static class ToMatConverter implements Converter<Matrix, Mat> {

        @Override
        public Mat convert(Matrix in) {
            final Mat result = new Mat(in.rows, in.cols, in.dt.mapping);

            int idx = 0;
            for (int r = 0 ; r < in.rows ; r++) {
                for (int c = 0; c < in.cols; c++) {
                    result.put(r, c, in.data[idx++]);
                }
            }

            return result;
        }

        @Override
        public JavaType getInputType(TypeFactory typeFactory) {
            return typeFactory.constructType(new TypeReference<Matrix>() {});
        }

        @Override
        public JavaType getOutputType(TypeFactory typeFactory) {
            return typeFactory.constructType(new TypeReference<Mat>() {});
        }
    }

    public static class Matrix {
        int rows;
        int cols;
        MatrixDataType dt;
        double[] data;

        public int getRows() {
            return rows;
        }

        public Matrix setRows(int rows) {
            this.rows = rows;
            return this;
        }

        public int getCols() {
            return cols;
        }

        public Matrix setCols(int cols) {
            this.cols = cols;
            return this;
        }

        public MatrixDataType getDt() {
            return dt;
        }

        public Matrix setDt(MatrixDataType dt) {
            this.dt = dt;
            return this;
        }

        public double[] getData() {
            return data;
        }

        public Matrix setData(double[] data) {
            this.data = data;
            return this;
        }

        double at(int x, int y) {
            if (x >= cols || y >= rows) throw new IllegalArgumentException("Bad coordinate");
            return data[y*rows + x];
        }

        @Override
        public String toString() {
            return "Matrix{" +
                    "rows=" + rows +
                    ", cols=" + cols +
                    ", dt=" + dt +
                    ", data=" + Arrays.toString(data) +
                    '}';
        }
    }
/*
    public static class MatDeserializer extends StdDeserializer<Mat> {

        protected MatDeserializer() {
            super(Mat.class);
        }

        @Override
        public Mat deserialize(JsonParser p, DeserializationContext ctxt) throws IOException, JsonProcessingException {
            final int rows, cols;
            final MatrixDataType dtype;
            final double[] data;
        }
    }

    public static class MatSerializer extends StdSerializer<Mat> {

        protected MatSerializer() {
            super(Mat.class);
        }

        @Override
        public void serialize(Mat value, JsonGenerator gen, SerializerProvider provider) throws IOException {
            gen.writeNumberField("rows", value.rows());
            gen.writeNumberField("cols", value.cols());
            gen.writeFieldName("data");
            gen.writeStartArray();
            gen.writeEndArray();
        }
    }
*/
    public enum MatrixDataType {
        d(CvType.CV_64F),
        f(CvType.CV_32F);

        public final int mapping;
        MatrixDataType(int mapping) {
            this.mapping = mapping;
        }
    }
}

Hope it helps,

EDIT:

I'm sorry, but I've not found reliable ways to:

• Choose array style. It appears to be an open issue on Jackson Github
• As for YAML labels, I've not found any clue about that. Maybe you'll have to fill an issue on the Jackson, if you decide to stick with it. After looking for a way to encode

EDIT 2: I"ve modified above code example to serialize/deserialize Mat objects. However, I've used loops as you did to fill/get matrix values, as I was not able to find a way to transfer values through ByteBuffers (and believe me, I tried. But neither Java API nor documentation are very helpful in this regard).

You can see that the introduced Matrix object is still around, becaus it eases conversion work IMHO. You can, if you want, get rid of it if you use Jackson StdSerializer object instead of Converter. However, I'm not sure that it would be cleaner.

Last words:

• I've found this other SO post about OpenCV matrix while searching for data transfer. Maybe it will be useful to you.
• Beware, my code only manage properly 64bit floating point values. You'll have to start from here to add other CvType cases if needed.

Well, this time it is sure, I cannot help you any further. Good luck ;-)