3

I have a large CSV file filled with data, the first 3 values in the row is unnecessary and the rest filled with values for a line, and many lines that would represent an image. Example X,X,X,1,2,3,4,5 X,X,X,6,7,8,9,10 X,X,X,11,12,13,14,15

I am trying to reconstruct it as an grayscale bitmap that i can display in a picturebox and export (as long with lots of other adjustments that I won't go into here as that's not the problem)

I have been struggling for the past 2 days to solve this problem. I have tried to directly parse the CSV in to Image.SetPixel and I had countless problems. I tried loading the CSV into a 2D array and I had issues in getting the length of the array and problems going out of bounds to the point where I scrapped both attempts as trying to rectify the problem made a tangled mess of code.

Does anyone have experience in tackling such a problem? Ever done it before? have any advice or code snippets? It would be very appreciated, Needless to say I am not the best at this.

Nyerguds
  • 5,360
  • 1
  • 31
  • 63
  • Would be good to attach sample csv to the question. – Evk Mar 20 '18 at 05:54
  • It's a massive file and i don't see a place to upload a file – user3599976 Mar 20 '18 at 06:12
  • The example is good enough IMO. It shows how the data is formatted, and _can_ in fact be used as 5x3 image. – Nyerguds Mar 20 '18 at 06:41
  • _I have tried to directly parse the CSV in to Image.SetPixel and I had countless problems._ Well what problems could that be? It is the 1st line of attack, a bit slow but fine to get a result to check the data. Later you can move to Lockbits for speed.. Of course you must get the parsing right and also know wat those numbers are supposed to mean! – TaW Mar 20 '18 at 08:30
  • 1
    @TaW Yea, the lack of any code is a bit iffy, but I would do this with 8-bit anyway, hence why I already answered. Though the only obvious problems I can see is if some of the rows are incomplete... otherwise the split result of the very first line should give the image width, and the amount of lines would obviously be the height... – Nyerguds Mar 20 '18 at 08:32

1 Answers1

1

This looks like the kind of thing to do with an 8-bit image. The advantage of an 8-bit image is that you simply need to have a one-dimensional byte array with your values, and you can pretty much load that straight into an image, using LockBits and Marshal.Copy.

Well, almost straight. The values of your pixels on an 8-bit image aren't actually your colors. They are references to the color palette, which actually contains your color. But if you want the values 0 to 255 on your image to refer to colors (0,0,0) to (255,255,255), then all you need to do is generate a palette with those 256 gray colors you want, which can be handled in one simple for-loop.

But first, the CSV. If this is really just simple "number, comma, number" information, you can simply use String.Split, but for any more advanced/reliable CSV parsing that can handle the special cases of quoted blocks containing quotes and/or the split character, you're going to need TextFieldParser. More info on that can be found here, though I think for this we can just go for the String.Split solution.

I made startColumn variable here for convenience, but in your case it'll of course be "3".

public static Bitmap GrayImageFromCsv(String[] lines, Int32 startColumn, Int32 maxValue)
{
    // maxValue cannot exceed 255
    maxValue = Math.Min(maxValue, 255);
    // Read lines; this gives us the data, and the height.
    //String[] lines = File.ReadAllLines(path);
    if (lines == null || lines.Length == 0)
        return null;
    Int32 bottom = lines.Length;
    // Trim any empty lines from the start and end.
    while (bottom > 0 && lines[bottom - 1].Trim().Length == 0)
        bottom--;
    if (bottom == 0)
        return null;
    Int32 top = 0;
    while (top < bottom && lines[top].Trim().Length == 0)
        top++;
    Int32 height = bottom - top;
    // This removes the top-bottom stuff; the new array is compact.
    String[][] values = new String[height][];
    for (Int32 i = top; i < bottom; i++)
        values[i - top] = lines[i].Split(',');
    // Find width: maximum csv line length minus the amount of columns to skip.
    Int32 width = values.Max(line => line.Length) - startColumn;
    if (width <= 0)
        return null;
    // Create the array. Since it's 8-bit, this is one byte per pixel.
    Byte[] imageArray = new Byte[width*height];
    // Parse all values into the array
    // Y = lines, X = csv values
    for (Int32 y = 0; y < height; y++)
    {
        Int32 offset = y*width;
        // Skip indices before "startColumn". Target offset starts from the start of the line anyway.
        for (Int32 x = startColumn; x < values[y].Length; x++)
        {
            Int32 val;
            // Don't know if Trim is needed here. Depends on the file.
            if (Int32.TryParse(values[y][x].Trim(), out val))
                imageArray[offset] = (Byte) Math.Max(0, Math.Min(val, maxValue));
            offset++;
        }
    }
    // generate gray palette for the given range, by calculating the factor to multiply by.
    Double mulFactor = 255d / maxValue;
    Color[] palette = new Color[maxValue + 1];
    for (Int32 i = 0; i <= maxValue; i++)
    {
        // Away from zero rounding: 2.4 => 2 ; 2.5 => 3
        Byte g = (Byte)Math.Round(i * mulFactor, MidpointRounding.AwayFromZero);
        palette[i] = Color.FromArgb(g, g, g);
    }
    // Since the palette is incomplete, give the color fill arg as Color.White
    return BuildImage(imageArray, width, height, width, PixelFormat.Format8bppIndexed, palette, Color.White);
}

Called as:

String[] lines = File.ReadAllLines(path);
using (Bitmap img = GrayImageFromCsv(lines, 3, 15))
{
    // null = conversion failed. Could log/show warning.
    if (img != null)
        img.Save("fromcsv.png", ImageFormat.Png);
}

The BuildImage function called at the end of the main processing does the aforementioned "loading byte array into image" operation. It can be found here. Note that the "stride" is the amount of bytes on one line of the image. While this is identical to the width for an 8-bit image, since each byte is one pixel, it will differ for other formats.

Nyerguds
  • 5,360
  • 1
  • 31
  • 63
  • First thank you for your response i'm sure it will be very helpful, but trying it out it seems to not know what to do with "ImageUtils" in your return, is this part of a library? – user3599976 Mar 20 '18 at 20:48
  • looking online it appears to only need 'System.Drawing' and ' System.Drawing.Imaging' both of which i have so im quite confused why its saying it does not exist in the current context – user3599976 Mar 21 '18 at 02:33
  • @user3599976 Read the last paragraph. I linked to the function (I edited it a bit now to make the link more noticeable). In my personal code repository it is situated in my static tool class `ImageUtils`, but obviously you can put it wherever you want and change/remove that `ImageUtils` part. – Nyerguds Mar 21 '18 at 08:06
  • Thank you very much, after modification to fit my data ranges and to add manual offset and multiplier it works fantastically! – user3599976 Mar 21 '18 at 18:47
  • You know, you could actually do that multiplying on the palette. If, for example, you got a range of 0 to 50, then you could leave your data as being only 0-50, and instead make sure the 0-50 palette indices are the full black to white fade, simply by taking larger steps when filling the palette with grey values. Then you only need to process those, and not every piece of data you receive. It has the added advantage that your original data is 100% preserved in the image. – Nyerguds Mar 21 '18 at 22:46
  • that's actually quite a good idea and would help a few issues i have been having with my current method, if i can figure out how to do it, this whole image processing thing is new to me. – user3599976 Mar 22 '18 at 02:06
  • Well, it's basically a matter of dividing 255 by the maximum value using floating point division, so you end up with a very accurate number and not just an integer to multiply with. I added the code. – Nyerguds Mar 22 '18 at 07:51
  • i actually figured it out where, something in my brain did not click 0x100 = 255, but thank you for this much cleaner version, you have been a real help – user3599976 Mar 22 '18 at 19:50
  • Oh but there is one thing im having a problem with, when trying to display img in a picturebox i get System.ArgumentException 'Parameter not valid' in Program.cs on execution, my solution is to read it from the file using stream (to not lock the file) and i feel like that's causing unnecessary lag am i missing something simple? – user3599976 Mar 22 '18 at 20:34
  • Well, 0x100 is 256 actually, but the original palette loop ran as long as the value was _smaller than_ 0x100, so, up to 0xFF, aka, 255. – Nyerguds Mar 22 '18 at 20:35
  • Ah, there are some oddities concerning loading images from streams. Specifically, you need to leave the stream open or the image will corrupt. I posted [a full bullet list on notes on that here a while ago.](https://stackoverflow.com/a/48579791/395685). To properly load the image without any such limitations you should load it from inside the using blocks for both the memstream and the image from that stream, as shown there, and then somehow make a true clean clone of the image in there. I usually use [my trusty CloneImage function](https://stackoverflow.com/a/48170549/395685) for that. – Nyerguds Mar 22 '18 at 20:44