A better way to implement collection of array of different types

Question

I'm looking for a semi-general purpose data structure in C# to store arrays of different integer and float types. In some cases, the integers are bit fields where each bit is equally important and loss of precision isn't tolerable. I'm finding this difficult and messy because of the C# type system and my lack of C# fluency.

The project: Ethercat periodic packets arrive and are converted to a structure (Packet) and accumulated as Packet[] over the course of an experiment. Each field of Packet from Packet[] is converted into an array.

I believe I'm looking for a way to 'wrap' these arrays into a single type so they can be part of a collection. Wrapping them has some other advantages (naming, hardware to SI scale factors, etc) to facilitate decoupling the hardware from the later implementation.

My best 'wrapper' is called 'DataWrapper' (simplified below) but with it I've made uncomfortable compromises in storage, loss of precision, use of object and quantity of code.

Is there a 'better' way in C#? My gold standard is the apparently trivial implementation without obvious compromises in Python using list of lists or numpy.arrays.

Could 'object' be used? How? Is it possible to box the whole array or must each array element boxed individually (inefficient)?

I've seen A list of multiple data types? however, seems like a lot of code and advanced programming techniques for what is essentially a List of List.

public class DataWrapper
{
    private double[] double_array;  // backing with double, but it could if I don't use float 
    private string name;
    private double scale_factor_to_SI;

    public DataWrapper(string name, double scale_factor, dynamic dynamic_array)
    {

        this.name = name;
        this.scale_factor_to_SI = scale_factor;
        this.double_array = new double[dynamic_array.Length];

        for (int cnt = 0; cnt < dynamic_array.Length; cnt++)
        {
            this.double_array[cnt] = (double)dynamic_array[cnt];
        }
    }

    public void Get(out int[] i_array)
    {
        i_array = this.double_array.Select(item => (int)item).ToArray();
    }

    public void Get(out long[] i_array)
    {
        i_array = this.double_array.Select(item => (long)item).ToArray();
    }

    public double[] GetSI()
    {
        return this.double_array.Select(item => this.scale_factor_to_SI * (double)item).ToArray();
    }
}

public struct Packet  // this is an example packet - the actual packet is much larger and will change over time.  I wish to make the change in 1 place not many.
{
    public long time_uS;
    public Int16 velocity;
    public UInt32 status_word;
};

public class example
{
    public Packet[] GetArrayofPacketFromHardware()
    {
        return null;
    }

    public example() {
        Packet[] array_of_packet = GetArrayofPacketFromHardware();

        var time_uS = array_of_packet.Select(p => p.time_uS).ToArray();
        var velocity = array_of_packet.Select(p => p.velocity).ToArray();
        var status_bits = array_of_packet.Select(p => p.status_word).ToArray();

        List<DataWrapper> collection = new List<DataWrapper> { };
        collection.Add(new DataWrapper("time", 1.0e-6, time_uS));
        collection.Add(new DataWrapper("velocity", 1/8192, velocity));
        collection.Add(new DataWrapper("status", 1, status_bits));  
    }
}

Answer 1

You could treat this a list of byte[] and serialize the value types using BitConverter to convert the value type to byte[] and then unroll it using the reverse call;

List<byte[]> dataList = new List<byte[]>();
float v = 1.0424f;
byte[] converted = BitConverter.GetBytes(v);
// put converted into a List<byte[]> 
dataList.Add(converted);
// Convert it back again
float z= BitConverter.ToSingle(dataList[0], 0);

Answer 2

floats are going to be problematic when sustained precision is required in C# ... period. That is unfortunate because we all know how much we like precision. But I don't think C# is the only language that suffers from this malady. That said, I think there is a way to achieve what you want and your wrapper is a good start.

I am not familiar with what you are using (3rd party library) so I'll just stick to offering a solution to the question.

If you know what the type is that you are retrieving, I would recommend using byte[]. This way you could effectively store 3 byte[] in a single list.

var dataList = new List<byte[]>();
dataList.Add(ConvertUsTime(p.time_US));
dataList.Add(ConvertVelocity(p.velocity));
dataList.Add(ConvertStatus(p.status));

byte[] ConvertToArray(long usTime) {
    return BitConverter.GetBytes(usTime);
}

byte[] ConvertVelocity(Int16 velocity) {
    return BitConverter.GetBytes(velocity);
}

byte[] ConvertStatus(UInt32 status) {
    return BitConverter.GetBytes(status);
}

... for a more generic method:

byte[] ConvertValue<T>(T value) where T : struct {
    // we have to test for type of T and can use the TypeCode for switch statement
    var typeCode = Type.GetTypeCode(typeof(T));

    switch(typeCode) {
        case TypeCode.Int64:
            return BitConverter.GetBytes((long)value);

        case TypeCode.Int16:
            return BitConverter.GetBytes((Int16)value);

        case TypeCode.UInt32:
            return BitConverter.GetBytes((UInt32)value);
    }

    return null;
}

Answer 3

Can you simply serialize the data to JSON or MessagePack and store it as an Array of strings? Seems like that would be relatively straightforward to implement and easy to use.

Answer 4

As a counter example to the generic list approach I would want to mention that the list example linked in the question should not be considered advanced. It uses an interface which is trivial C#.

Using different types implementing the same interface can be a better solution when you expect you want to debug the contents of the list or expect the business logic on the different types of collections to grow. Right now you only have GetSI() but it might grow with more generic methods that have specific implementations for each packet collection type. Finally, Debugging lists containing generic objects or raw bytes is probably not very well supported by your IDEs debugger. Interfaces are well supported. An implementation to illustrate the idea is shown below.

public example() {

    Packet[] array_of_packet = GetArrayofPacketFromHardware();

    var time_uS = array_of_packet.Select(p => p.time_uS).ToArray();
    var velocity = array_of_packet.Select(p => p.velocity).ToArray();
    var status_bits = array_of_packet.Select(p => p.status_word).ToArray();

    List<IPacketCollection> collection = new List<IPacketCollection> { };
    collection.Add(new TimePacketCollection(time_uS));
    collection.Add(new VelocityPacketCollection(velocity));
    collection.Add(new StatusPacketCollection(status_bits));  

    // Now we have benefits over generic objects or byte arrays.
    // We can extend our collections with additional logic as your 
    // Application grows or right now already still benefit from the 
    // GetSI you mentioned as a plus in your question.
    foreach(var velocityPacketCollection in collection.OfType<VelocityPacketCollection>()) {
        // specific velocity collection things here.
        // Also your debugger is perfectly happy peeking in the collection.
    }

    // or generic looping accessing the GetSI()
    foreach(var packetCollection in collection) {
        System.Debug.Println(packetCollection.GetSI());
    }
}

public interface IPacketCollection {
    /// <summary>
    /// Not sure what this method should do but it seems it 
    /// always returns double precision or something?
    /// </summary>
    public double[] GetSI;
} 

public class TimePacketCollection : IPacketCollection {
    private const double scaleFactor = 1.0e-6;
    private long[] timePacketArray;

    public TimePacketCollection(long[] timeArray) {
        timePacketArray = timeArray;
    }

    public double[] GetSI(){
         // no IDE available. Not sure if this automatically converts to 
         // double due to multiplication with a double.
         return timePacketArray.Select(item => scaleFactorToSI * item).ToArray();
    }
}

public class VelocityPacketCollection : IPacketCollection {
    private const double scaleFactor = 1/8192;
    private Int16[] velocityPacketArray;

    public VelocityPacketCollection (Int16[] velocities) {
        velocityPacketArray = velocities;
    }

    public double[] GetSI(){
         // no IDE available. Not sure if this automatically converts to 
         // double due to multiplication with a double.
         return velocityPacketArray.Select(item => scaleFactorToSI * item).ToArray();
    }
}

public class StatusPacketCollection : IPacketCollection {
    private const double scaleFactor = 1.0;
    private UInt32[] statusPacketArray;

    public StatusPacketCollection (UInt32[] statuses) {
        statusPacketArray = statuses;
    }

    public double[] GetSI(){
         // no IDE available. Not sure if this automatically converts to 
         // double due to multiplication with a double.
         return statusPacketArray.Select(item => scaleFactorToSI * item).ToArray();
    }
}

Disclaimer: I wrote this from a device without an IDE. I am absolutely disastrous writing code without my IDE correcting me from stupid mistakes so if this doesn't compile please bear with me. I think the generic idea is clear.

Answer 5

The answer to "Is there a better way in C#?" - is yes.

Use List<dynamic> as the collection for your arrays.

List<dynamic> array_list = new List<dynamic> { };
public void AddArray(dynamic dynamic_array)
{
   this.array_list.Add(dynamic_array);
}

Of course anything could be passed into it - but that can be tested for.

List<dynamic> is better in this situation than ArrayList as when attempting to index an array taken from 'list' the IDE flags an error.

int ndx = 0;
foreach (var array_from_list in this.array_list) {                
  var v = array_from_list[ndx];  // error if array_from_list is ArrayList
}

A complete illustration follows (but it only conceptually replicates my above wrapper).

using System;
using System.Collections.Generic;


namespace Application
{
    class MyTest
    {
        List<dynamic> array_list = new List<dynamic> { };
        int length;

        public void AddArray(dynamic dynamic_array)
        {
            this.array_list.Add(dynamic_array);
            this.length = dynamic_array.Length;
        }
        public dynamic GetVector(int ndx)
        {
            return array_list[ndx];
        }
        public void Display()
        {
            for (int ndx = 0; ndx < this.length; ndx++)
            {
                string ln_txt = "";
                foreach (var array_from_list in this.array_list)
                {
                    string s = array_from_list[ndx].ToString();
                    ln_txt += $"{s} ";
                }

                Console.WriteLine(ln_txt);
            }

        }
    }

    static class Program
    {


        [STAThread]
        static void Main(string[] args)
        {

            MyTest test = new MyTest();
            test.AddArray(new long[] { 10, 20, 30, 40 });
            test.AddArray(new int[] { 1, 2, 3, 4 });
            test.AddArray(new double[] { .1, .2, .3, .4 });
            test.AddArray(new string[] { "a", "b", "c", "d" });
            test.Display();


            for (int vecnum = 0; vecnum < 4; vecnum++)
            {
                var vector = test.GetVector(vecnum);
                Console.Write($"vnum:{vecnum} :   ");

                foreach (var value in vector)
                {
                    Console.Write($"{value}   ");
                }
                Console.WriteLine("");
            }

        }
    }
}

I've since learned that https://stackoverflow.com/a/10380448/4462371 is probably more correct technical explanation.