Some operations (Convolve, AffineTransform) on BufferedImage with 16 bit per channel result in garbled data

Question

Some operations on BufferedImages with 16 bit per channel result in images with random colored pixels. Is it possible to avoid this problem?

I see the problem at least with

• ConvolveOp
• AffineTransformOp with INTERPOLATION_BICUBIC on images with alpha channel

Sample code:

Kernel kernel = new Kernel(2, 2, new float[] { 0.25f, 0.25f, 0.25f, 0.25f });
ConvolveOp blurOp = new ConvolveOp(kernel, ConvolveOp.EDGE_NO_OP, null);
img = blurOp.filter(img, null);

Input: output image:

The operations work fine when the image is 8 bit per channel.

I tried to convert the image from 16 to 8 bit per channel while keeping the color profile using the following code but this also results in a garbled image.

private static BufferedImage changeTo8BitDepth(BufferedImage bi) {
    ColorModel cm = bi.getColorModel();
    boolean hasAlpha = cm.hasAlpha();
    boolean isAlphaPre = cm.isAlphaPremultiplied();
    int transferType = DataBuffer.TYPE_BYTE;
    int transparency = cm.getTransparency();
    ColorSpace cs = cm.getColorSpace();
    ColorModel newCm = new ComponentColorModel(cs, hasAlpha, isAlphaPre, transparency, transferType);
    WritableRaster newRaster = newCm.createCompatibleWritableRaster(bi.getWidth(), bi.getHeight());
    BufferedImage newBi = new BufferedImage(newCm, newRaster, isAlphaPre, null);
    // convert using setData
    newBi.setData(bi.getRaster());
    return newBi;
}

(It is possible to use ColorConvertOp to convert to an 8-bit sRGB image but I need the non-sRGB color profile.)

I tested on Java 8, 11, and 17 on macOS and Linux. For full source code and images for tests see https://github.com/robcast/java-imaging-test (class Test16BitColor)

Answer 1

After som testing and research, I think the fact that ConvolveOp and AffineTransformOp doesn't work with 16 bits/sample (TYPE_USHORT data type) images out of the box, is a JDK bug. It might be that the underlying native code only works with 8 bits/sample images, but in that case "Op"s should throw an exception (or perhaps add a slower, but correct Java fallback code path). You might want to report that to the OpenJDK community.

---

For the 16 to 8 bits/sample conversion, the problem is you can't set 16 bit values into an 8 bit buffer, as there's no normalization done on the samples. I guess you'll just end up with the lower 8 bits of the 16 bits sample, which will typically look like static/noise. This can be fixed, however.

Here's a version that will convert the values correctly to 8 bit, but otherwise keep the color space/color profile unchanged:

private static BufferedImage changeTo8BitDepth(BufferedImage original) {
    ColorModel cm = original.getColorModel();

    // Create 8 bit color model
    ColorModel newCM = new ComponentColorModel(cm.getColorSpace(), cm.hasAlpha(), cm.isAlphaPremultiplied(), cm.getTransparency(), DataBuffer.TYPE_BYTE);
    WritableRaster newRaster = newCM.createCompatibleWritableRaster(original.getWidth(), original.getHeight());
    BufferedImage newImage = new BufferedImage(newCM, newRaster, newCM.isAlphaPremultiplied(), null);

    // convert using createGraphics/dawImage
    Graphics2D graphics = newImage.createGraphics();
    try {
        graphics.drawImage(original, 0, 0, null);
    }
    finally {
        graphics.dispose();
    }

    return newImage;
}

If you prefer conversion using rasters only, it's also possible with some hacks:

private static BufferedImage changeTo8BitDepth(BufferedImage original) {
    ColorModel cm = original.getColorModel();

    // Create 8 bit color model
    ColorModel newCM = new ComponentColorModel(cm.getColorSpace(), cm.hasAlpha(), cm.isAlphaPremultiplied(), cm.getTransparency(), DataBuffer.TYPE_BYTE);
    WritableRaster newRaster = newCM.createCompatibleWritableRaster(original.getWidth(), original.getHeight());
    BufferedImage newImage = new BufferedImage(newCM, newRaster, newCM.isAlphaPremultiplied(), null);

    // convert using setData
    // newImage.setData(as8BitRaster(original.getRaster())); // Works
    newRaster.setDataElements(0, 0, as8BitRaster(original.getRaster())); // Faster, requires less conversion

    return newImage;
}

private static Raster as8BitRaster(WritableRaster raster) {
    // Assumption: Raster is TYPE_USHORT (16 bit) and has PixelInterleavedSampleModel
    PixelInterleavedSampleModel sampleModel = (PixelInterleavedSampleModel) raster.getSampleModel();

    // We'll create a custom data buffer, that delegates to the original 16 bit buffer
    final DataBuffer buffer = raster.getDataBuffer();

    return Raster.createInterleavedRaster(new DataBuffer(DataBuffer.TYPE_BYTE, buffer.getSize()) {
        @Override public int getElem(int bank, int i) {
            return buffer.getElem(bank, i) >>> 8; // We only need the upper 8 bits of the 16 bit sample
        }

        @Override public void setElem(int bank, int i, int val) {
            throw new UnsupportedOperationException("Raster is read only!");
        }
    }, raster.getWidth(), raster.getHeight(), sampleModel.getScanlineStride(), sampleModel.getPixelStride(), sampleModel.getBandOffsets(), new Point());
}

Answer 2

I had a similar problem with blurring a 16 bit BufferedImage and ConvolveOp, but in my case a grayscale picture (BufferedImage.TYPE_BYTE_GRAY). The image I needed to process is a heightmap.

I used the following code:

private static BufferedImage blur(BufferedImage img) {
    float[] matrix3 = {
            1f/9, 1f/9, 1f/9,
            1f/9, 1f/9, 1f/9,
            1f/9, 1f/9, 1f/9,
        };
    Kernel kernel = new Kernel(3, 3, matrix3);

    BufferedImageOp op = new ConvolveOp(kernel);
    return op.filter(img, null);
}

This results in the following artifacts (images cropped due to their large size):

Original image:

Blurred image:

Now, instead of using ConvolveOp, I can achieve the same effect with blurring all rows and all cols individually in only one dimension (cf. here) with a factor of 1/3 instead of 1/9 in the matrix, because 1/3 * 1/3 = 1/9. The following code does this, but leads to the same artifacts as the ConvolveOp used above:

private static BufferedImage blur(BufferedImage img) {
    for (int x=0; x<img.getWidth(); x++) {
        for (int y=0; y<img.getHeight(); y++) {
            int y0 = Math.max(y-1, 0);
            int y1 = y;
            int y2 = Math.min(y+2, img.getHeight()-1);

            short s0 = ((short[]) img.getRaster().getDataElements(x, y0, null))[0];
            short s1 = ((short[]) img.getRaster().getDataElements(x, y1, null))[0];
            short s2 = ((short[]) img.getRaster().getDataElements(x, y2, null))[0];
            
            short sNew = (short) ((s0 + s1 + s2) / 3);

            img.getRaster().setDataElements(x, y, new short[] {(short)sNew});
        }
    }
    
    for (int x=0; x<img.getWidth(); x++) {
        for (int y=0; y<img.getHeight(); y++) {
            int x0 = Math.max(x-1, 0);
            int x1 = x;
            int x2 = Math.min(x+2, img.getWidth()-1);

            short s0 = ((short[]) img.getRaster().getDataElements(x0, y, null))[0];
            short s1 = ((short[]) img.getRaster().getDataElements(x1, y, null))[0];
            short s2 = ((short[]) img.getRaster().getDataElements(x2, y, null))[0];
            
            short sNew = (short) ((s0 + s1 + s2) / 3);

            img.getRaster().setDataElements(x, y, new short[] {(short)sNew});
        }
    }
    return img;
}

Workaround:

With the following code (added lines marked), I can get rid of these artifacts and the blurring is correct:

private static BufferedImage blur(BufferedImage img) {
    for (int x=0; x<img.getWidth(); x++) {
        for (int y=0; y<img.getHeight(); y++) {
            int y0 = Math.max(y-1, 0);
            int y1 = y;
            int y2 = Math.min(y+2, img.getHeight()-1);

            short s0 = ((short[]) img.getRaster().getDataElements(x, y0, null))[0];
            short s1 = ((short[]) img.getRaster().getDataElements(x, y1, null))[0];
            short s2 = ((short[]) img.getRaster().getDataElements(x, y2, null))[0];

            s0 += 32768; // ADDED LINE
            s1 += 32768; // ADDED LINE
            s2 += 32768; // ADDED LINE
            
            short sNew = (short) ((s0 + s1 + s2) / 3);
            sNew -= 32768; // ADDED LINE

            img.getRaster().setDataElements(x, y, new short[] {(short)sNew});
        }
    }
    
    for (int x=0; x<img.getWidth(); x++) {
        for (int y=0; y<img.getHeight(); y++) {
            int x0 = Math.max(x-1, 0);
            int x1 = x;
            int x2 = Math.min(x+2, img.getWidth()-1);

            short s0 = ((short[]) img.getRaster().getDataElements(x0, y, null))[0];
            short s1 = ((short[]) img.getRaster().getDataElements(x1, y, null))[0];
            short s2 = ((short[]) img.getRaster().getDataElements(x2, y, null))[0];

            s0 += 32768; // ADDED LINE
            s1 += 32768; // ADDED LINE
            s2 += 32768; // ADDED LINE
            
            short sNew = (short) ((s0 + s1 + s2) / 3);
            sNew -= 32768; // ADDED LINE

            img.getRaster().setDataElements(x, y, new short[] {(short)sNew});
        }
    }
    return img;
}

Final image:

It seems to me like it has to do something with the Java short type being signed, while the type of the BufferedImage is unsigned, but I'm not yet completely sure about that. Maybe this helps solving your problem.