C# get dominant color in an image

Question

I'm building a program that makes screenshots from a video. It extracts frames from the video (with ffmpeg) and then combines them into one file.

All works fine, except sometimes I get (almost) black images, mostly in the beginning and ending of the video.

A possible solution I can think of is to detect if the extracted frame is dark. If it is dark, extract another frame from a slightly different time.

How can I detect if the extracted frame is dark/black? Or is there another way I can solve this?

private void getScreenshots_Click(object sender, EventArgs e)
{
    int index = 0;
    foreach (string value in this.filesList.Items)
    {
        string file = selectedFiles[index] + "\\" + value;

        // ------------------------------------------------
        //   MediaInfo
        // ------------------------------------------------
        // https://github.com/Nicholi/MediaInfoDotNet
        //
        // get file width, height, and frame count
        //
        // get aspect ratio of the video
        // and calculate height of thumbnail 
        // using width and aspect ratio
        //
        MediaInfo MI = new MediaInfo();
        MI.Open(file);
        var width = MI.Get(StreamKind.Video, 0, "Width");
        var height = MI.Get(StreamKind.Video, 0, "Height");
        decimal d = Decimal.Parse(MI.Get(StreamKind.Video, 0, "Duration"));
        decimal frameCount = Decimal.Parse(MI.Get(StreamKind.Video, 0, "FrameCount"));
        MI.Close();
        decimal ratio = Decimal.Divide(Decimal.Parse(width), Decimal.Parse(height));
        int newHeight = Decimal.ToInt32(Decimal.Divide(newWidth, ratio));
        decimal startTime = Decimal.Divide(d, totalImages);
        //totalImages - number of thumbnails the final image will have
        for (int x = 0; x < totalImages; x++)
        {
            // increase the time where the thumbnail is taken on each iteration
            decimal newTime = Decimal.Multiply(startTime, x);
            string time = TimeSpan.FromMilliseconds(double.Parse(newTime.ToString())).ToString(@"hh\:mm\:ss");

            string outputFile = this.tmpPath + "img-" + index + x + ".jpg";

            // create individual thumbnails with ffmpeg
            proc = new Process();
            proc.StartInfo.FileName = "ffmpeg.exe";
            proc.StartInfo.Arguments = "-y -seek_timestamp 1 -ss " + time + " -i \"" + file + "\" -frames:v 1 -qscale:v 3 \"" + outputFile + "\"";
            proc.Start();
            proc.WaitForExit();
        }

        // set width and height of final image
        int w = (this.cols * newWidth) + (this.spacing * this.cols + this.spacing);
        int h = (this.rows * newHeight) + (this.spacing * this.rows + this.spacing);

        int left, top, i = 0;
        // combine individual thumbnails into one image
        using (Bitmap bmp = new Bitmap(w, h))
        {
            using (Graphics g = Graphics.FromImage(bmp))
            {
                g.Clear(this.backgroundColor);
                // this.rows - number of rows
                for (int y = 0; y < this.rows; y++)
                {
                    // put images on a column
                    // this.cols - number of columns
                    // when x = number of columns go to next row
                    for (int x = 0; x < this.cols; x++)
                    {
                        Image imgFromFile = Image.FromFile(this.tmpPath + "img-" + index + i + ".jpg");
                        MemoryStream imgFromStream = new MemoryStream();
                        imgFromFile.Save(imgFromStream, imgFromFile.RawFormat);
                        imgFromFile.Dispose();

                        left = (x * newWidth) + ((x + 1) * this.spacing);
                        top = (this.spacing * (y + 1)) + (newHeight * y);
                        g.DrawImage(Image.FromStream(imgFromStream), left, top, newWidth, newHeight);
                        i++;
                    }
                }
            }

            // save the final image
            bmp.Save(selectedFiles[index] + "\\" + value + ".jpg");
        }
        index++;
    }
}

score 0 · Accepted Answer · edited May 23 '17 at 12:08

You would need to analyze the individual frames. Loop through the pixels of each frame/image and calculate the overall brightness.

This is actually outlined in a SO post about determining overall brightness of an image. You could apply the same idea to your needs.

Depending on how you are wanting to implement this, it could be rather processing heavy. If this needs to be done on the fly, you would most likely want to narrow down your frame analysis to smaller portions instead of the entire image. If, however, you are doing this as more of a post-processing technique, then it may not matter as much.

@CK13 Great. If that answered your question, could you accept this as the answer to close the question out? Thanks. — gmiley, Jan 22 '17 at 15:09

score 0 · Answer 2 · edited May 23 '17 at 12:16

Thanks to gmiley i managed to solve my problem (hopefully).

Here's how:

I included the function linked by gmiley in my class
I added a function that takes an image and converts it to bitmap, found here

I moved the code that created the thumbnails to its own function

private void ffmpegGetThumb(string input, string output, double time, int getNewFrame = 0)
{
    // create individual thumbnails with ffmpeg
    string timeString = TimeSpan.FromMilliseconds(time).ToString(@"hh\:mm\:ss");
    proc = new Process();
    proc.StartInfo.FileName = "ffmpeg.exe";
    proc.StartInfo.Arguments = "-y -seek_timestamp 1 -ss " + timeString + " -i \"" + input + "\" -frames:v 1 -qscale:v 3 \"" + output + "\"";
    proc.Start();
    proc.WaitForExit();
}

I created a function that converts the thumbnail to bitmap, checks its brightness and adjusts the time at which the thumbnail is taken, if needed

private void brightThumb(string input, string output, double time, int getNewFrame, bool goBack)
{
    //get initial thumbnail
    this.ffmpegGetThumb(input, output, time);
    Bitmap frame = this.ConvertToBitmap(output);
    double brightness = CalculateAverageLightness(frame);
    double newTime = time;

    if (CalculateAverageLightness(frame) < 0.1)
    {
        // brightness is to low
        // get a new thumbnail by increasing the time at which it's taken
        if (getNewFrame == 1 && goBack == false)
        {
            newTime = time + 1000;
        }
        // if goBack is true (the time is beyond 75% of the total duration)
        // we decrease the time (if curent time is at credits no point going forward)
        else if (getNewFrame == 1 && goBack == true)
        {
            newTime = time - 1000;
        }
        // take a new thumbnail at the adujted time
        this.brightThumb(input, output, newTime, 1, goBack);
    }
}

Calling the function, where the code for creating the thumbs was in the getScreenshots_Click()

bool goBack = false;
if (x > (totalImages/4) * 3)
{
    goBack = true;
}
this.brightThumb(file, outputFile, time, 0, goBack);

Performance wise i see no big change in execution time compared to before doing this checks. If it finds many dark images then it will retake them so this will obviously increase the execution time.

There is one problem I can think of. If the video ends with a dark scene long enough to include 2 thumbnails. In this case the thumbnails will be the same, since it will go back one second until it finds an image that is bright enough.

C# get dominant color in an image

2 Answers2