What is the best way to store a sbyte[,] to disk and back

Question

I want to store an sbyte[,] on my disk using as little space as possible (cant take more than a few seconds to save or load though) and get it back at a later time.

I can't serialize it to xml: Cannot serialize object of type System.SByte[,]. Multidimensional arrays are not supported.

And I can't convert it to a MemoryStream: Cannot convert from 'sbyte[,]' to 'int'

Besides creating a text file and looping it out piece by piece.. what options are there.

If it makes any difference the array can be upwards of 100 000 x 100 000 in size. The file also needs to be usable by different operating systems and computers.

---

Update.

I went with flattening my array down to a 1D sbyte[] and then converted the sbyte[] to a stream and saving it to disk along with a separate file containing the dimensions.

Stream stream = new MemoryStream(byteArray);

Used this as a base for saving the stream to disk. https://stackoverflow.com/a/5515894/937131

This is a testcase I wrote for the flattening and unflattening if anyone else finds it usefull.

[TestMethod]
public void sbyteTo1dThenBack()
{
    sbyte[,] start = new sbyte[,]
    {
        {1, 2},
        {3, 4},
        {5, 6},
        {7, 8},
        {9, 10}
    };

    sbyte[] flattened = new sbyte[start.Length];
    System.Buffer.BlockCopy(start, 0, flattened, 0, start.Length * sizeof(sbyte));

    sbyte[,] andBackAgain = new sbyte[5, 2];
    Buffer.BlockCopy(flattened, 0, andBackAgain, 0, flattened.Length * sizeof(sbyte));

    var equal =
        start.Rank == andBackAgain.Rank &&
        Enumerable.Range(0, start.Rank).All(dimension => start.GetLength(dimension) == andBackAgain.GetLength(dimension)) &&
        andBackAgain.Cast<sbyte>().SequenceEqual(andBackAgain.Cast<sbyte>());

    Assert.IsTrue(equal);
}

Answer 1

As per my comments, I feel that writing out the byte array equivalents of everything is the way to go here. This may not be the most efficient way to do it, and lacks a lot of error handling code that you will need to supply, but, it works in my tests.

Edit: Also, BitConverter.ToInt32() may depend on the "Endianness" of your processor. See Scott Chamberlain's comments on how to fix this if you intend to use this code on ARM or other non-x86 systems.

public static class ArraySerializer
{
    public static void SaveToDisk(string path, SByte[,] input)
    {
        var length = input.GetLength(1);
        var height = input.GetLength(0);
        using (var fileStream = File.OpenWrite(path))
        {
            fileStream.Write(BitConverter.GetBytes(length), 0, 4);//Store the length
            fileStream.Write(BitConverter.GetBytes(height), 0, 4);//Store the height
            var lineBuffer = new byte[length];
            for (int h = 0; h < height; h++) 
            {
                for (int l = 0; l < length; l++) 
                {
                    unchecked //Preserve sign bit
                    {
                        lineBuffer[l] = (byte)input[h,l];
                    }
                }
                fileStream.Write(lineBuffer,0,length);
            }

        }
    }
    public static SByte[,] ReadFromDisk(string path)
    {
        using (var fileStream = File.OpenRead(path))
        {
            int length;
            int height;
            var intBuffer = new byte[4];
            fileStream.Read(intBuffer, 0, 4);
            length = BitConverter.ToInt32(intBuffer, 0);
            fileStream.Read(intBuffer, 0, 4);
            height = BitConverter.ToInt32(intBuffer, 0);
            var output = new SByte[height, length]; //Note, for large allocations, this can fail... Would fail regardless of how you read it back
            var lineBuffer = new byte[length];
            for (int h = 0; h < height; h++)
            {
                fileStream.Read(lineBuffer, 0, length);
                for (int l = 0; l < length; l++)
                    unchecked //Preserve sign bit
                    {
                        output[h,l] = (SByte)lineBuffer[l];
                    }
            }
            return output;
        }
    }
}

Here's how I tested it:

void Main()
{
    var test = new SByte[20000, 25000];
    var length = test.GetLength(1);
    var height = test.GetLength(0);
    var lineBuffer = new byte[length];
    var random = new Random();
    //Populate with random data
    for (int h = 0; h < height; h++) 
    {
        random.NextBytes(lineBuffer);
        for (int l = 0; l < length; l++)
        {
            unchecked //Let's use first bit as a sign bit for SByte
            {
                test[h,l] = (SByte)lineBuffer[l];
            }
        }
    }
    var sw = Stopwatch.StartNew();
    ArraySerializer.SaveToDisk(@"c:\users\ed\desktop\test.bin", test);
    Console.WriteLine(sw.Elapsed);
    sw.Restart();
    var test2 = ArraySerializer.ReadFromDisk(@"c:\users\ed\desktop\test.bin");
    Console.WriteLine(sw.Elapsed);
    Console.WriteLine(test.GetLength(0) == test2.GetLength(0));
    Console.WriteLine(test.GetLength(1) == test2.GetLength(1));
    Console.WriteLine(Enumerable.Cast<SByte>(test).SequenceEqual(Enumerable.Cast<SByte>(test2))); //Dirty hack to compare contents... takes a very long time
}

On my system (with an SSD), that test takes ~2.7s to write or read the contents of the 20kx25k array. To add compression, you can just wrap the FileStream in a GZipStream.