Libtiff: Generating TIFF/EP Profile 2 Raw image from pixel array

Question

I'm trying to generate a TIFF/EP Profile 2 raw (.dng) image from an arbitrary pixel array. This pixel array represents a Bayer pattern (CFA).

I studied the TIFF/EP file specifications and, by using libtiff, I included all tags I believe are necessary to generate a complete .dng file. However, I cannot convert the created file with dcraw (dcraw displays it cannot decode the file).

One problem might be given by two tags which are declared as mandatory in the TIFF/EP specification but don't seem to be implemented in libtiff: The SensingMethod and TIFF/EPStandardID tags. Do I really have to include them (I have seen example code which neglects these tags but is still reported to work properly), and if so, how can I manually add them to libtiff? Moreover, setting the SubIFD tag yields the error message "Assertion failed: *pa<=0xFFFFFFFFUL, file tif_dirwrite.c, line 1869"

All in all, I don't think my problems are just due to these three tags and I believe there is something fundamentally wrong. Maybe someone of you can have a look on my code and give some clues? I have to say that the documentation of libtiff is rather poor, so my code is inspired by one of the very few example codes out there: elphel_dng.c.

Thank you very much! Fabian

PS. I uploaded the generated file on Dropbox

C++

#include "tiffio.h"
#include <iostream>
using namespace std;

int main(void)
{
TIFF *tif = TIFFOpen("8bitRaw.dng", "w");

const int sampleperpixel = 1;
const int width = 4;
const int height = 4;

static const short bayerPatternDimensions[] = { 2,2 };
static const double cam_xyz[] = { 2.0413690, -0.5649464, -0.3446944, -0.9692660, 1.8760108, 0.0415560,0.0134474, -0.1183897, 1.0154096 }; //AdobeRGB
static const double neutral[] = { 1.0, 1.0, 1.0 };
long max_white = 255;
long sub_offset = 0;
int row, i;
float gamma = 80;

//Arbitrary Bayer pixel values:
unsigned char image[height][width*sampleperpixel] = {};
image[0][0] = 5;    image[0][1] = 165;   image[0][2] = 0;   image[0][3] = 255;
image[1][0] = 0;    image[1][1] = 21;    image[1][2] = 0;   image[1][3] = 10;
image[2][0] = 0;    image[2][1] = 0;     image[2][2] = 30;  image[2][3] = 5;
image[3][0] = 21;   image[3][1] = 120;   image[3][2] = 1;   image[3][3] = 254;

//Black Thumbnail pixel values:
unsigned char Thumbnail_RGB_Array[] = { 0,0,0,0,0,0,0,0,0,0,0,0 };

//Linearization Table:
unsigned short curve[256];
for (i = 0; i < 256; i++)
    curve[i] = 255 * pow(i / 255.0, 100.0 / gamma) + 0.5;


TIFFSetField(tif, TIFFTAG_SUBFILETYPE, 1);
TIFFSetField(tif, TIFFTAG_IMAGEWIDTH, 4);
TIFFSetField(tif, TIFFTAG_IMAGELENGTH, 4);
TIFFSetField(tif, TIFFTAG_BITSPERSAMPLE, 8);
TIFFSetField(tif, TIFFTAG_SAMPLESPERPIXEL, 3);
TIFFSetField(tif, TIFFTAG_ORIENTATION, ORIENTATION_TOPLEFT);
TIFFSetField(tif, TIFFTAG_COMPRESSION, COMPRESSION_NONE);
TIFFSetField(tif, TIFFTAG_PLANARCONFIG, PLANARCONFIG_CONTIG);
TIFFSetField(tif, TIFFTAG_PHOTOMETRIC, PHOTOMETRIC_RGB);
TIFFSetField(tif, TIFFTAG_XRESOLUTION, 75.0);
TIFFSetField(tif, TIFFTAG_YRESOLUTION, 75.0);
TIFFSetField(tif, TIFFTAG_RESOLUTIONUNIT, RESUNIT_INCH);

TIFFSetField(tif, TIFFTAG_MAKE, "DummyMake");
TIFFSetField(tif, TIFFTAG_MODEL, "DummyModel");
TIFFSetField(tif, TIFFTAG_SOFTWARE, "DummySoftware");
TIFFSetField(tif, TIFFTAG_ORIGINALRAWFILENAME, 1, "DummyName.dng");
TIFFSetField(tif, TIFFTAG_UNIQUECAMERAMODEL, "DummyUniqueModel");
TIFFSetField(tif, TIFFTAG_IMAGEDESCRIPTION, "DummyImageDescription");
TIFFSetField(tif, TIFFTAG_COPYRIGHT, "DummyCopyright");
TIFFSetField(tif, TIFFTAG_DATETIME, "2016:06:30 11:11:15");
TIFFSetField(tif, TIFFTAG_DNGVERSION, "\01\01\00\00");
TIFFSetField(tif, TIFFTAG_DNGBACKWARDVERSION, "\01\00\00\00");
TIFFSetField(tif, TIFFTAG_COLORMATRIX1, 9, cam_xyz);
TIFFSetField(tif, TIFFTAG_ASSHOTNEUTRAL, 3, neutral);
TIFFSetField(tif, TIFFTAG_CALIBRATIONILLUMINANT1, 21);
//SensingMethodTag and TIFF/EPStandardID tag: Libtiff doesn't seem to know these:
//TIFFSetField(tif, 37399, 2);
//TIFFSetField(tif, 37398, "\01\01\00\00");
//Yields an error:
//TIFFSetField(tif, TIFFTAG_SUBIFD, 1, &sub_offset);

//Write a black 4x4 pixel dummy thumbnail:
for (row = 0; row < 4 ; row++) 
    TIFFWriteScanline(tif, Thumbnail_RGB_Array, row, 0);
TIFFWriteDirectory(tif);


//Now write main raw image:
TIFFSetField(tif, TIFFTAG_SUBFILETYPE, 0);
TIFFSetField(tif, TIFFTAG_IMAGEWIDTH, width);
TIFFSetField(tif, TIFFTAG_IMAGELENGTH, height);
TIFFSetField(tif, TIFFTAG_BITSPERSAMPLE, 8);
TIFFSetField(tif, TIFFTAG_PHOTOMETRIC, PHOTOMETRIC_CFA);
TIFFSetField(tif, TIFFTAG_SAMPLESPERPIXEL, 1);
TIFFSetField(tif, TIFFTAG_PLANARCONFIG, PLANARCONFIG_CONTIG);
TIFFSetField(tif, TIFFTAG_CFAREPEATPATTERNDIM, bayerPatternDimensions);
TIFFSetField(tif, TIFFTAG_CFAPATTERN, "\01\00\02\01");
TIFFSetField(tif, TIFFTAG_CFAPLANECOLOR, 3, "\00\01\02");
TIFFSetField(tif, TIFFTAG_LINEARIZATIONTABLE, 256, curve);
TIFFSetField(tif, TIFFTAG_WHITELEVEL, 1, &max_white);

TIFFWriteScanline(tif, image[0], 0, 0);
TIFFWriteScanline(tif, image[1], 1, 0);
TIFFWriteScanline(tif, image[2], 2, 0);
TIFFWriteScanline(tif, image[3], 3, 0);
TIFFClose(tif);
return 0;
}

@mark Yeah, sure. [Here](https://www.dropbox.com/s/b1garp645eoj64q/8bitRaw.dng?dl=0) — Fabian K, Jul 04 '16 at 12:19
It's a TIFF/EP Profile 2 image (at least I hope libtiff has created that kind of file format). In the file [specifications] (http://standardsproposals.bsigroup.com/Home/getPDF/567)), it says: "The file extension ".dng" shall specify a Profile 2 file." That's not to confuse with Adobe's DNG. @MarkSetchell — Fabian K, Jul 04 '16 at 12:27
Are you sure `dcraw` can read your *"funny TIFs"*? I thought `dcraw` would be used for reading `Adobe DNG` files since they are Digital Negatives and it is used for reading Adobe DNG's produced by high-end cameras. If you change your file extension to `TIF` ImageMagick can read them fine. — Mark Setchell, Jul 04 '16 at 12:47
Really, can it? Thanks for the info, I'll have a closer look into that. Actually, the TIFF/EP Profile 2 format is for storing raw data, i.e. pixel intensities, Bayer pattern layout, etc. And since there is a link to the file specifications on the dcraw website, I guess dcraw is capable of dealing with it... By the way, Adobe's DNG is based on TIFF/EP. @MarkSetchell — Fabian K, Jul 04 '16 at 13:05
What `dcraw` command are you actually trying to use? And what do you actually ant to get as a result? — Mark Setchell, Jul 05 '16 at 08:48
@MarkSetchell: I simulate a camera via tracing rays through an objective onto the camera sensor. The problem comes down to the following: I have a pixel array of intensities (the image on my camera sensor) and the layout of the Bayer pattern. With these two information, I want to generate a raw image (just as if a real camera would generate one). dcraw (or any other raw converter) shall be able to open that image file so that I can apply demosaicing algorithms etc... I call dcraw on windows via "dcraw64.exe imagefile.dng" (I also tried imagefile.tiff). Thank you for your continuous help :) — Fabian K, Jul 05 '16 at 21:05
You have gone past the limits of my knowledge, I am afraid. I know Eric (@emcconville) is very knowledgable on TIFF files, so maybe he will have a chance to take a look now he is alerted. — Mark Setchell, Jul 06 '16 at 08:04

Vanista · Answer 1 · 2019-04-24T17:07:43.293

I have pretty much the same requirement and continued from where you left off. Here's a bit of code that generates a 128x128 raw Bayer 10 bits DNG image that is compatible with dng_validate, dcraw, lightroom and resolve.

I've stripped out almost everything and put the minimum to keep dng_validate happy.

#include <tiffio.h>
#include <math.h>

int main (void)
{
   const int width = 128;
   const int height = 128;

   static const short bayerPatternDimensions[] = { 2, 2 };
   static const float ColorMatrix1[] =
   {
      1.0, 0.0, 0.0,
      0.0, 1.0, 0.0,
      0.0, 0.0, 1.0,
   };

   static const float AsShotNeutral[] =
   {
      1.0, 1.0, 1.0,
   };

   int row, i;
   float gamma = 80;

   #define GR 0x80, 0x20, 0x08, 0x02, 0x00,
   #define WH 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
   #define BL 0x00, 0x00, 0x00, 0x00, 0x00,
   #define BLK(N) N N N N N N N N

   // Arbitrary Bayer pixel values:
   unsigned char image[] =
   {
      BLK(WH)
      BLK(GR)
      BLK(BL)
      BLK(WH)
      BLK(GR)
      BLK(BL)
      BLK(WH)
      BLK(GR)
      BLK(BL)
   };

   TIFF* tif = TIFFOpen ("out.dng", "w");
   TIFFSetField (tif, TIFFTAG_DNGVERSION, "\01\01\00\00");
   TIFFSetField (tif, TIFFTAG_SUBFILETYPE, 0);
   TIFFSetField (tif, TIFFTAG_COMPRESSION, COMPRESSION_NONE);
   TIFFSetField (tif, TIFFTAG_IMAGEWIDTH, width);
   TIFFSetField (tif, TIFFTAG_IMAGELENGTH, height);
   TIFFSetField (tif, TIFFTAG_BITSPERSAMPLE, 10);
   TIFFSetField (tif, TIFFTAG_ROWSPERSTRIP, 1);
   TIFFSetField (tif, TIFFTAG_ORIENTATION, ORIENTATION_TOPLEFT);
   TIFFSetField (tif, TIFFTAG_PHOTOMETRIC, PHOTOMETRIC_CFA);
   TIFFSetField (tif, TIFFTAG_SAMPLESPERPIXEL, 1);
   TIFFSetField (tif, TIFFTAG_PLANARCONFIG, PLANARCONFIG_CONTIG);
   TIFFSetField (tif, TIFFTAG_SAMPLEFORMAT, SAMPLEFORMAT_UINT);
   TIFFSetField (tif, TIFFTAG_CFAREPEATPATTERNDIM, bayerPatternDimensions);
   TIFFSetField (tif, TIFFTAG_CFAPATTERN, "\00\01\01\02");
   TIFFSetField (tif, TIFFTAG_MAKE, "DNG");
   TIFFSetField (tif, TIFFTAG_UNIQUECAMERAMODEL, "DNG");
   TIFFSetField (tif, TIFFTAG_COLORMATRIX1, 9, ColorMatrix1);
   TIFFSetField (tif, TIFFTAG_ASSHOTNEUTRAL, 3, AsShotNeutral);
   TIFFSetField (tif, TIFFTAG_CFALAYOUT, 1);
   TIFFSetField (tif, TIFFTAG_CFAPLANECOLOR, 3, "\00\01\02");

   unsigned char* cur = image;

   for (row = 0; row < height;)
   {
      for (i = 0; i < 32; ++i, ++row)
         TIFFWriteScanline (tif, cur, row, 0);
      cur += 40;
   }

   TIFFClose (tif);
   return 0;
}

dcraw was pretty stubborn to get working, I had to get its source and trace in gdb to find the faulty condition:

if (!load_raw || height < 22 || width < 22 ||
  tiff_bps > 16 || tiff_samples > 6 || colors > 4)
  is_raw = 0;

So your first attempt at creating a 4x4 image would automatically fail because of this.

Notice: this code doesn't work. The generated file doesn't even have the correct size when displayed by dcraw -i. — , Apr 20 '17 at 22:18
If you change `i < 40` to `i < 32` in the for loop condition, it will then write the correct number of lines to produce a 128x128 image. Thanks for this, I used it to write a Python library for creating dng files. — Amanda_Panda, Sep 06 '18 at 01:44
For modern libtiff versions, you must provide the size for CFAPattern. eg: `TIFFSetField (tif, TIFFTAG_CFAPATTERN, 4, "\00\01\01\02");` — Shootfast, Aug 14 '21 at 19:33
If the CFAPATTERN line segfaults, it's because of the missing `4, ` in front of the string, as Shootfast says. This is the case as of libtiff5 version 4.4.0. If I'm not mistaken, the API change was in [version 4.2.0](http://libtiff.maptools.org/v4.2.0.html). — user2394284, Oct 19 '22 at 10:00

Libtiff: Generating TIFF/EP Profile 2 Raw image from pixel array

1 Answers1