I think basically @Dan Mašek already answered the question in the comment section.
I will try to summarize the findings for jpg files as an answer and I am glad about any improvements.
CMYK to Grayscale
If you want to convert your jpg file from CMYK we have to look into grfmt_jpeg.cpp. There exist other files like this for different image codes.
Depending on the numbers of color channels cinfo is assigned. For CMYK images the cinfo is set to 4 and the function on line 504 icvCvt_CMYK2Gray_8u_C4C1R is called.
This function can be found in utils.cpp:
void icvCvt_CMYK2Gray_8u_C4C1R( const uchar* cmyk, int cmyk_step,
uchar* gray, int gray_step, Size size )
{
int i;
for( ; size.height--; )
{
for( i = 0; i < size.width; i++, cmyk += 4 )
{
int c = cmyk[0], m = cmyk[1], y = cmyk[2], k = cmyk[3];
c = k - ((255 - c)*k>>8);
m = k - ((255 - m)*k>>8);
y = k - ((255 - y)*k>>8);
int t = descale( y*cB + m*cG + c*cR, SCALE );
gray[i] = (uchar)t;
}
gray += gray_step;
cmyk += cmyk_step - size.width*4;
}
}
and uses fixed variables for the conversion:
#define SCALE 14
#define cR (int)(0.299*(1 << SCALE) + 0.5)
#define cG (int)(0.587*(1 << SCALE) + 0.5)
#define cB ((1 << SCALE) - cR - cG)
RGB/BGR to Grayscale
If your image only contains three color channels it seems that libjpeg is used for the conversion. This can be seen in line 717. (I am not 100% sure if this is the correct line).
In jdcolor.c it can be seen that there a definitions and standards for converting color channels starting from line 41.
The most important part for your specific question is:
the conversion equations to be implemented are therefore
R = Y + 1.402 * Cr
G = Y - 0.344136286 * Cb - 0.714136286 * Cr
B = Y + 1.772 * Cb
Y = 0.299 * R + 0.587 * G + 0.114 * B
which relate to standards of the ITU-R and are used in many other sources I found.
More detailed information can be found here and here.
The second source relating to a StackOverflow question makes it clear that the conversion does not only depend on the pure RGB values but also on other parameters as gamma value.
The standard OpenCV uses seems to be Rec. 601.
