7

Assume I have a large csv file that I'd like to read, modify and write back. Perhaps I want to change a field separator from comma to tab. Perhaps I want to change the quotation character from ' to ". Or may be I want to add a plus sign to each value in the first column. Since the file is large, I do not want to load it in memory all at once, I'd like to read it record by record.

So I write a code like this:

var inPath = @"infile.txt";
var outPath = @"outfile.txt";

CsvConfiguration readCf = GetReadConfiguration();
CsvConfiguration writeCf = GetWriteConfiguration();

using (var streamin = new FileStream(inPath, FileMode.Open))
using (var streamReader = new StreamReader(streamin))
{
    using (var csvReader = new CsvReader(streamReader, readCf))
    using (var csvWriter = new CsvWriter(new StreamWriter(outPath), writeCf))
    {
        while (csvReader.Read())
        {
            var currentRecord = csvReader.GetRecord<dynamic>();
            UpdateRecord(currentRecord);
            csvWriter.WriteRecord(currentRecord);
        }
    }
}

This fails at run time with the following error:

Types that inherit IEnumerable cannot be auto mapped. Did you accidentally call GetRecord or WriteRecord which acts on a single record instead of calling GetRecords or WriteRecords which acts on a list of records?

Note, that nothing interesting happens in UpdateRecord, in fact this line above can be completely commented out.

What happens is that GetRecord returns an ExpandoObject and then WriteRecord chokes on this object.

What is the best way to make this work?

Update Just so it's clear: I fully realize that I'm getting this error is because CSVHelper does not support ExpandoObjects for WriteRecord call. I'm looking for suggestions, for easiest way to make this work.

Andrew Savinykh
  • 25,351
  • 17
  • 103
  • 158
  • The error is telling you to use GetRecords (plural) instead of singular. I mean it is hinting it. Did you try that? – CodingYoshi Dec 02 '16 at 22:56
  • @CodingYoshi, nope, that's not what I need. – Andrew Savinykh Dec 02 '16 at 22:57
  • Look at Mar L's answer [here](http://stackoverflow.com/questions/33294738/read-all-values-from-csv-into-a-list-using-csvhelper) and see if that helps you. – CodingYoshi Dec 02 '16 at 23:01
  • @CodingYoshi, thank you, no it does not. The answer suggests not using CSVHelper. I need CSVHelper because this is a non trivial CSV file. It has quoted fields, some quoted fields are multi-line, etc. CSVHelper is perfect for parsing this file and writing it back, if only we could get around this deficiency. – Andrew Savinykh Dec 02 '16 at 23:05
  • If all you are doing is changing CSV characteristics (like ticks etc) rather than the data, couldnt you just read the raw fields (untyped) – Ňɏssa Pøngjǣrdenlarp Dec 02 '16 at 23:24
  • @Plutonix, sorry, could you explain that? That is I do not understand how reading raw untyped fields is different from what I'm doing. – Andrew Savinykh Dec 02 '16 at 23:29
  • Basically, its what Mr Close suggests in the linked post. Rather than mapping it to any object, get the lines as a string array. The link may have changed since neither answer suggests not using CSVHelper – Ňɏssa Pøngjǣrdenlarp Dec 02 '16 at 23:34
  • @Plutonix, well that accounts for the reading part, put then I'll have to format the result for writing manually. Thank you for the suggestion. – Andrew Savinykh Dec 02 '16 at 23:37
  • Yes, there appears to be no way to pass Writer an array - a loop on the fields I guess. – Ňɏssa Pøngjǣrdenlarp Dec 02 '16 at 23:41

1 Answers1

4

It appears that ability to write ExpandoObject is added in this commit.

Note that this is not in the latest nuget release (2.16.3 as of the time of writing) but for the next 3.x release. If it's an option for you get latest beta from github and use that.

If you do, note that you will have to call NextRecord after each WriteRecord. In 2.16.3 CVSHelper would call it for you, so it's a bit of a breaking API change

Andrew Savinykh
  • 25,351
  • 17
  • 103
  • 158
  • 1
    The old version gave me `CsvConfigurationException: Types that inherit IEnumerable cannot be auto mapped. Did you accidentally call GetRecord or WriteRecord which acts on a single record instead of calling GetRecords or WriteRecords which acts on a list of records?` and the new version (3.0) gives me `CsvTypeConverterException: Converting IEnumerable types is not supported for a single field. If you want to do this, create your own ITypeConverter and register it in the TypeConverterFactory by calling AddConverter.` – Nikolay Kostov Jun 26 '17 at 09:19