0

I am running a cron job in C# that takes 200,000 images and convert into 1 bit image. During running this job, sometimes the process crashes (even though I have global try catch), sometimes for some images(not all) it throws OutOfMemoryException and sometimes for some images it it throws A generic error occurred in GDI.

        int pageSize = 1000;
        for (int pageNumber = 0; pageNumber < 200; pageNumber++)
        {
            var imageUrls = allIMageUrls.Skip(pageSize * pageNumber).Take(pageSize).ToList();
            var counter = 0;
            var total = imageUrls.Count;
            Logger.Log($"Page Number : {pageNumber}");
            var failedImageUrls = new System.Collections.Concurrent.ConcurrentBag<string>();
            Parallel.ForEach(imageUrls, imageUrl =>
            {
                try
                {
                    Interlocked.Increment(ref counter);
                    var image = _httpService.DownloadImage(imageUrl);
                    if (image != null && image.Length > 0)
                    {
                        var oneBitImage = ConvertToOnebitFaxGroup4(contract);
                        _httpService.UploadImage(image, oneBitImage);
                        oneBitImage = null;
                        image  = null;
                    }
                }
                catch (Exception ex)
                {
                    failedImageUrls.Add(imageUrl);                        
                    Logger.Log(ex);
                }
            });

This is one time process. I added paging so that when it crashes I can restart from that page instead of start at beginning.

 public static class ImageProcessor
 {

    static ImageCodecInfo _codecInfo;
    static EncoderParameters _encoderParameters;

    static ImageProcessor()
    {
        foreach (var codec in ImageCodecInfo.GetImageEncoders())
        {
            if (codec.MimeType == "image/tiff")
            {
                _codecInfo = codec;
                break;
            }
        }
        _encoderParameters = new EncoderParameters(2);
        _encoderParameters.Param[0] = new EncoderParameter(Encoder.Compression, (long)EncoderValue.CompressionCCITT4);
        _encoderParameters.Param[1] = new EncoderParameter(Encoder.ColorDepth, (long)1);
    }

    public static byte[] ConvertToOnebitFaxGroup4(byte[] bytes)
    {
        using (var memoryStream = new MemoryStream(bytes))
        {
            var image = Image.FromStream(memoryStream);
            var pData = Marshal.AllocHGlobal(bytes.Length);
            Marshal.Copy(bytes, 0, pData, bytes.Length);
            var bytesPerLine = (image.Width + 31) / 32 * 4; 
            var img = new Bitmap(image.Width, image.Height, bytesPerLine, PixelFormat.Format1bppIndexed, pData);
            using (var ms = new MemoryStream())
            {
                image.Save(ms, _codecInfo, _encoderParameters);
                img.Dispose();
                Marshal.FreeHGlobal(pData);
                return ms.ToArray();
            }
        }
    }

Updated:

    public static byte[] ConvertToOnebitFaxGroup4(byte[] bytes)
    {
        using (var memoryStream = new MemoryStream(bytes))
        {
            using (var image = Image.FromStream(memoryStream))
            {
                var pData = Marshal.AllocHGlobal(bytes.Length);
                Marshal.Copy(bytes, 0, pData, bytes.Length);
                var bytesPerLine = (image.Width + 31) / 32 * 4;
                using (var img = new Bitmap(image.Width, image.Height, bytesPerLine, PixelFormat.Format1bppIndexed, pData))
                {
                    using (var ms = new MemoryStream())
                    {
                        img.Save(ms, _codecInfo, _encoderParameters);
                        Marshal.FreeHGlobal(pData);
                        return ms.ToArray();
                    }
                }
            }
        }
    }

Update2

    public static byte[] ConvertToOnebitFaxGroup4(byte[] bytes)
    {
        using (var memoryStream = new MemoryStream(bytes))
        {
            using (var image = Image.FromStream(memoryStream))
            {
                using (var ms = new MemoryStream())
                {
                    image.Save(ms, _codecInfo, _encoderParameters);
                    return ms.ToArray();
                }
            }
        }
    }
Imran Qadir Baksh - Baloch
  • 32,612
  • 68
  • 179
  • 322
  • You forgot to dispose the Image and Bitmap objects. GDI objects are expensive and limited. Another strange thing is copying the bytes around like this, Marshal.AllocHGlobal, the multiple memory streams. Format conversion in GDI+ is performed using ImageEncoder and ImageDecorer objects – Panagiotis Kanavos Sep 13 '18 at 13:04
  • u mean I should put image and img(Bitmap) into using blocks or call explicitly dispose? That code was copied using https://stackoverflow.com/questions/26930656/convert-tiff-to-1bit. Do you have any suggestion to improve the above code. – Imran Qadir Baksh - Baloch Sep 13 '18 at 13:07
  • Or you can just call [Image.Save](https://learn.microsoft.com/en-us/dotnet/api/system.drawing.image.save?view=netframework-4.7.2) with a target stream and format info. [How to: Convert a BMP image to a PNG image](https://learn.microsoft.com/en-us/dotnet/framework/winforms/advanced/how-to-convert-a-bmp-image-to-a-png-image) shows that `Save`can accept a specific format, or more detailed coded, encoder parameters – Panagiotis Kanavos Sep 13 '18 at 13:07
  • 2
    Image, Bitmap objects *have* to be disposed. Yes, you need `using` blocks for all of them. Most of the code though can be deleted. `Image.Save` can save to a target stream and use a different format. You could write `image.Save(ms,codecInfo,encoderParameters)` to create a new image. *Don't* reuse the `image` variable for both the source and target image. Both of these have to be disposed at the end – Panagiotis Kanavos Sep 13 '18 at 13:08
  • BTW I am disposing bitmap. Can you please answer the question with code that can be simplified. – Imran Qadir Baksh - Baloch Sep 13 '18 at 13:13
  • I have found a bug in the code. I think it should `img.Save` instead of `image.Save(ms`? – Imran Qadir Baksh - Baloch Sep 13 '18 at 13:33
  • Can you please see the updated code? – Imran Qadir Baksh - Baloch Sep 13 '18 at 13:40
  • You don't even need `img`, it's never used. Once you remove it, `pdata` isn't needed either, or any of the Marhal. code. `image` should be declared in a `using` statement to ensure it's disposed. – Panagiotis Kanavos Sep 13 '18 at 13:41
  • img is Bitmap, I needed to specify the 1 bit image? – Imran Qadir Baksh - Baloch Sep 13 '18 at 13:43
  • You specified that in the encoder parameters – Panagiotis Kanavos Sep 13 '18 at 13:43
  • Updated code again. Please check. Thanks for your help – Imran Qadir Baksh - Baloch Sep 13 '18 at 13:47
  • 1
    Can you please answer this? So that it will be helpful for others – Imran Qadir Baksh - Baloch Sep 13 '18 at 14:05
  • Just wanna to confirm after making this change. No more crash, no exception of any type. You are genious. :) – Imran Qadir Baksh - Baloch Sep 13 '18 at 14:21

0 Answers0