How to create video using avcodec from jpeg images of type OpenCV::Mat?

Question

I have colored jpeg images of OpenCV::Mat type and I create from them video using avcodec. The video that I get is upside-down, black & white and each row of each frame is shifted and I got diagonal line. What could be the reason for such output? Follow this link to watch the video I get using avcodec. I'm using acpicture_fill function to create avFrame from cv::Mat frame!

P.S. Each cv::Mat cvFrame has width=810, height=610, step=2432 I noticed that avFrame (that is filled by acpicture_fill) has linesize[0]=2430 I tried manually setting avFrame->linesizep0]=2432 and not 2430 but it still didn't helped.

======== CODE =========================================================

AVCodec *encoder = avcodec_find_encoder(AV_CODEC_ID_H264);
AVStream *outStream = avformat_new_stream(outContainer, encoder);
avcodec_get_context_defaults3(outStream->codec, encoder);

outStream->codec->pix_fmt = AV_PIX_FMT_YUV420P;
outStream->codec->width = 810;
outStream->codec->height = 610;
//...

SwsContext *swsCtx = sws_getContext(outStream->codec->width, outStream->codec->height, PIX_FMT_RGB24,
                                    outStream->codec->width, outStream->codec->height,  outStream->codec->pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);

for (uint i=0; i < frameNums; i++)
{
    // get frame at location I using OpenCV
    cv::Mat cvFrame;
    myReader.getFrame(cvFrame, i); 
    cv::Size frameSize = cvFrame.size();    
    //Each cv::Mat cvFrame has  width=810, height=610, step=2432


1.  // create AVPicture from cv::Mat frame
2.  avpicture_fill((AVPicture*)avFrame, cvFrame.data, PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
3avFrame->width = frameSize.width;
4.  avFrame->height = frameSize.height;

    // rescale to outStream format
    sws_scale(swsCtx, avFrame->data, avFrame->linesize, 0, outStream->codec->height, avFrameRescaledFrame->data, avFrameRescaledFrame ->linesize);
encoderRescaledFrame->pts=i;
avFrameRescaledFrame->width = frameSize.width;
    avFrameRescaledFrame->height = frameSize.height;

av_init_packet(&avEncodedPacket);
    avEncodedPacket.data = NULL;
    avEncodedPacket.size = 0;

    // encode rescaled frame
    if(avcodec_encode_video2(outStream->codec, &avEncodedPacket, avFrameRescaledFrame, &got_frame) < 0) exit(1);
    if(got_frame)
    {
        if (avEncodedPacket.pts != AV_NOPTS_VALUE)
            avEncodedPacket.pts =  av_rescale_q(avEncodedPacket.pts, outStream->codec->time_base, outStream->time_base);
        if (avEncodedPacket.dts != AV_NOPTS_VALUE)
            avEncodedPacket.dts = av_rescale_q(avEncodedPacket.dts, outStream->codec->time_base, outStream->time_base);

        // outContainer is "mp4"
        av_write_frame(outContainer, & avEncodedPacket);

        av_free_packet(&encodedPacket);
    }
}

UPDATED

As @Alex suggested I changed the lines 1-4 with the code below

int width = frameSize.width, height = frameSize.height; 
avpicture_alloc((AVPicture*)avFrame, AV_PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
for (int h = 0; h < height; h++)
{
     memcpy(&(avFrame->data[0][h*avFrame->linesize[0]]), &(cvFrame.data[h*cvFrame.step]), width*3);
}

The video (here) I get now is almost perfect. It's NOT upside-down, NOT black & white, BUT it seems that one of the RGB components is missing. Every brown/red colors became blue (in original images it should be vice-verse). What could be the problem? Could rescaling(sws_scale) to AV_PIX_FMT_YUV420P format causes this?

Alex I · Accepted Answer · 2012-12-02T09:07:32.770

2

The problem in a nutshell: avpicture_fill() expects no padding between rows, ie the stride (step) to be equal to width*sizeof(pixel), ie 810*3 = 2430. The actual stride of the data in cv::Mat step as you say is 2432 which is different, so just passing the data directly won't work. There is no way to tell avpicture_fill() to use a different stride for the input data; it is not part of the API (you might say it should be :)

There are two possible solutions:

Create an array in which the input data is contiguous, no padding between rows. You'd have to memcopy each row from the cv::Mat into that array. Then pass it to avpicture_fill().

int width, height; // get from mat
uint8_t* buf = malloc(width * height * 3); // 3 bytes per pixel
for (int i = 0; i < height; i++)
{
    memcpy( &( buf[ i*width*3 ] ), &( mat->data[ i*mat->step ] ), width*3 );
}
avpicture_fill(..., buf, ...)

Btw, to flip the video vertically, you can do this to copy the last row to the first and so forth:

...
    memcpy( &( buf[ i*width*3 ] ), &( mat->data[ (height - i - 1)*mat->step ] ), width*3 );
...

Or, fill in the AVPicture yourself:

AVPicture* pic = malloc(sizeof(AVPicture));
avpicture_alloc(pic, PIX_FMT_BGR24, width, height);
for (int i = 0; i < height; i++)
{
    memcpy( &( pic->data[0][ i*pic->linesize[0] ] ),  &( mat->data[ i*mat->step ] ), width*3);
}

There is no need to allocate pic->data[0] or set pic->linesize[0], avpicture_alloc() should do that. There is also no need to fill in data[1] or data[2], those should be null.

EDIT: Removed old code which showed copying R, G, B to separate planes. PIX_FMT_BGR24 is not a planar format.

I'm not familiar enough with OpenCV C++ API to figure out how to get the width and height (it's not mat->width, obviously) but I think you know what I mean.

P.S. Btw, your video is not actually black and white. It's just that each successive row is offset by two bytes, so the colors are rotated: red becomes green, green becomes blue, and so forth. The result is grayscale-ish, but if you look closely the individual rows are colored.

edited Dec 02 '12 at 09:07

answered Dec 01 '12 at 23:57

Alex I

19,689
9
86
158

I understand that I need to allocate buffer for `pic->data[0]` and for `pic->data[1]` and `pic->data[2]` equally to how much? and `pic->linesize[0]` is also 0, should I put there 2430? – theateist Dec 02 '12 at 08:04
I'm actually not quite sure how avpicture_fill() fills a picture when the format is BGR24. Please see the edit above. – Alex I Dec 02 '12 at 09:02
You don't need to allocate pic->data[0] or set linesize, avpicture_alloc() should do that. This is all for the second method anyway. Please try the first method first (copy data to buf without padding and call avpicture_fill). – Alex I Dec 02 '12 at 09:09
it's not BGR24, but RGB24. My mistake. I updated my post. I'll try the first method – theateist Dec 02 '12 at 10:14
I tried what you suggested and it almost succeeded. The problem is that every brown/red colors became blue (in original images it should be vice-verse). Do you have any ideas? I update my post and attached link to the video I get(https://picasaweb.google.com/103161760482140400348/2December201202#5817342583034854274) – theateist Dec 02 '12 at 14:00
@theateist: Good, getting close :) If you are calling avpicture_fill and sws_getContext with RGB format, call both with BGR instead (keeping everything else the same). If you are calling with BGR, try RGB. That will swap the red and blue. – Alex I Dec 02 '12 at 17:23
It helped! I changed both `avpicture_alloc` and `sws_getContext` to PIX_FMT_BGR24 and it helped, but why? Does this mean that the jpeg image was coded with BGR and not RGB? The second question is where did you read that `avpicture_fill` expects no padding between rows? – theateist Dec 02 '12 at 21:12
@theateist: jpeg is usually stored as YCbCr, but it does seem that when you create a cv::Mat from a jpeg the result is BGR. For avpicture_fill, [this doc](http://ffmpeg.org/doxygen/trunk/group__lavc__picture.html) says "always assume a linesize alignment of 1" which I interpreted as no padding. Also, there is no way to specify any particular linesize/padding when passing data to avpicture_fill, so no padding would be the default assumption. Please remember to accept my answer and upvote it :) Thanks! – Alex I Dec 03 '12 at 04:02
If I could I would give you a medal for your explanations! – theateist Dec 03 '12 at 12:31

score 0 · Answer 2 · edited May 23 '17 at 12:21

0

Have you considered using OpenCV's features to create the video for you? It's much more easier since your data is already store in a cv::Mat.

If you would like to keep your approach, you could simply rotate the cv::Mat.

edited May 23 '17 at 12:21

Community

1
1

answered Dec 01 '12 at 10:51

karlphillip

92,053
36
243
426

I cannot use OpenCV's to create video. I have to use avcodec for this. I tried to use rotate, but it didn't fixed it. I still get Black&White video and diagonal lines. I guess it something with stride – theateist Dec 01 '12 at 12:11

score 0 · Answer 3 · edited Jul 23 '15 at 13:05

0

About the color problem in the UPDATE of the original post. Is that caused by,

OpenCV Mat is (BGR) -> FFmpeg AVFrame is (RGB) ?

If so, try,

cvtColor( cvFrame , cvFrame , CV_BGR2RGB ) ;

before line 1.

edited Jul 23 '15 at 13:05

Baris Demiray

1,539
24
35

answered Sep 09 '13 at 04:00

maythe4thbewithu

387
1
2
12

How to create video using avcodec from jpeg images of type OpenCV::Mat?

3 Answers3