c# CSVHelper read CSV with variable headers

Question

First time using the csvReader - note it requires a custom class that defines the Headers found in the CSV file.

class DataRecord
{
    //Should have properties which correspond to the Column Names in the file 
    public String Amount { get; set; }
    public String InvoiceDate { get; set; }......
}

The example given then uses the class such:-

            using (var sr = new StreamReader(@"C:\\Data\\Invoices.csv"))
        {
            var reader = new CsvReader(sr);

            //CSVReader will now read the whole file into an enumerable
            IEnumerable<DataRecord> records = reader.GetRecords<DataRecord>();

            //First 5 records in CSV file will be printed to the Output Window
            foreach (DataRecord record in records.Take(5)) 
            {
                Debug.Print("{0} {1}, {2}", record.Amount, record.InvoiceDate, ....);
            }

Two questions :- 1. The app will be loading in files with differing headers so I need to be able to update this class on the fly - is this possible & how? (I am able to extract the headers from the CSV file.)

2. CSV file is potentially multi millions of rows (gb size) so is this the best / most efficient way of importing the file.

Destination is a SQLite DB - debug line is used as example.

Thanks

Answer 1

1. The app will be loading in files with differing headers so I need to be able to update this class on the fly - is this possible & how?

Although it is definetely possible with reflecion or third part libraries, creating an object for a row will be inefficient for such a big files. Moreover, using C# for such a scenario is a bad idea (unless you have some business data transformation). I would consider something like this, or perhaps a SSIS package.