filehelpers - Parsing variable line length

Question

I have to parse (C#) a .CSV file, with a variable "width" and 2 lines of header information (the fist one being name and the second one being the unit).

The data looks like:

Example1.CSV:

"timestamp","NAME_1","NAME_2","NAME_3","NAME_4"
"ms","unit_1","unit_2","unit_3","unit_4"
0.01,1.23,4.56,7.89,0.12
0.02,1.23,4.66,7.89,0.11
0.03,1.23,4.76,7.89,0.11
0.04,56.23,4.86,7.89,0.12

Example2.CSV:

"timestamp","NAME_1","NAME_2","NAME_3","NAME_4","NAME_5",...,"NAME_N"
"ms","unit_1","unit_2","unit_3","unit_4","unit_5",...,"unit_N"
0.01,1.23,4.56,7.89,0.12,0.13,...,0.27
0.02,1.23,4.66,7.89,0.12,0.13,...,0.22
0.03,1.23,4.76,7.89,0.11,0.13,...,0.24
0.04,56.23,4.86,7.89,0.12,0.13,...,0.29

With N being the "width" of the table (value can be up to 128 and larger). I'm planning to use Filehelpers.

I thought of using [FieldOptional()] - but that gets very unhandy, especially when the "width" is variable...

My current attempt looks like

[IgnoreFirst(2)]
[DelimitedRecord(",")]
public sealed class LogData
{

    public Double ts;

    public Double Field1;

    [FieldNullValue(0.0)]
    [FieldOptional()]
    public Double Field2;

    [FieldNullValue(0.0)]
    [FieldOptional()]
    public Double Field3;

    // and so on
}

Any help on "how to solve the variable width"-Problem in a more elegant manner is appreciated - Thank you very much in advance!

Ben

How are you planning to use the model once you've read in the CSV file? — JMK, Apr 16 '14 at 12:17
Model is used for plotting and filtering data. Either I'm working with the Array-Result from `LogData[] res = engine.ReadFile(@"..\\..\\testdata.csv");` or I'm converting it to a DataTable - I planning to let performance tests decide. — Boern, Apr 16 '14 at 12:25

score 2 · Accepted Answer · answered Apr 16 '14 at 12:45

2

If you are planning to convert the file into DataTable, there is a better option

Please use CsvEngine of FileHelpers library. See the code snippet below:

using (MemoryStream stream = new MemoryStream(_fileContent)) //file content can be file as byte array
            {
                TextReader reader = new StreamReader(stream);
string path = "C:\\Sample.csv";
                CsvEngine csvEngine = new CsvEngine("Model", ',', path);
                var dataTable = csvEngine.ReadStreamAsDT(reader);
//Do whatever with dataTable

}

Here the sample file can be csv file or text file contains the header of the csv file you want to process. The columns of the DataTable will be named according to the header of the sample file

Cheers

answered Apr 16 '14 at 12:45

Prasanth

3,029
31
44

Boombastico! Even the first line is automatically added as `dataTable.Columns[i].ColumnName`. Any Idea how I could store the column specific unit-information in the `DataTable` as well (second line in the .CSV)? Maybe abusing the `ColumnName` along with custom converter and a custom separator "|||"? – Boern Apr 16 '14 at 13:51
@Ben You can get the unit information in the second row of the CSV as the second row in the DataTable if the number of values are same as the number of columns. If you need the same in any custom format, you need to use some tweaks. – Prasanth Apr 16 '14 at 15:27
Sadly, that's no option, since I want to convert[1] the values to a `Double` `DataTable` afterwards (needed for working with the values plotting and processing). [1] http://stackoverflow.com/questions/9028029/how-to-change-datatype-of-a-datacolumn-in-a-datatable – Boern Apr 16 '14 at 16:22

score 2 · Answer 2 · answered Apr 23 '14 at 09:07

You can use optional array field. I think you need to be using FileHelpers 2.9.9.

[IgnoreFirst(2)]
[DelimitedRecord(",")]
public class LogData
{
    public Double TimeStamp;

    [FieldNullValue(0.0)]
    [FieldOptional, FieldArrayLength(0, 100)]
    public Double[] ManyFields;
}

Here's a working example.

class Program
{
    static String content = 
         @"""timestamp"",""NAME_1"",""NAME_2"",""NAME_3"",""NAME_4""
         ""ms"",""unit_1"",""unit_2"",""unit_3"",""unit_4""
         0.01,1.23,4.56,7.89,0.12
         0.02,1.23,4.66,7.89,0.11
         0.03,1.23,4.76,7.89,0.11
         0.04,56.23,4.86,7.89,0.12";

    private static void Main()
    {
        var engine = new FileHelperEngine<LogData>();
        var records = engine.ReadString(content);
        Assert.AreEqual(0.01, records[0].TimeStamp);
        Assert.AreEqual(1.23, records[0].ManyFields[0]);
        Assert.AreEqual(4.56, records[0].ManyFields[1]);
        Assert.AreEqual(7.89, records[0].ManyFields[2]);
        Assert.AreEqual(0.12, records[0].ManyFields[3]);
    }
}

filehelpers - Parsing variable line length

2 Answers2

Linked