How do I strip \n characters from a file?

Question

I have a question that should make most people go "WTF?", but I have it nonetheless.

I've got a bunch of data files from a vendor. It's in a custom flat-file format that claims to be CSV, except it's not comma separated, and values are not quoted. So, not really CSV at all.

foo,bar,baz
alice,bob,chris

And so on, except much longer and less interesting. The problem is, some records have embedded newlines (!!!):

foo,bar
rab,baz
alice,bob,chris

That is supposed to be two records of three fields each. Normally, I would just say "No, this is stupid.", but I inadvisedly looked closer, and discovered that it was actually a different kind of end of line than the actual line ending sequence:

foo,bar\n
rab,baz\r\n
alice,bob,chris\r\n

Note the \n on the first line. I've determined that this holds for all the cases I found of embedded newlines. So, I need to basically do s/\n$// (I tried this specific command, it did not do anything).

Note: I don't actually care about the contents of the fields, so replacing a newline with nothing is fine. I just need each line in the file to have the same number of records (ideally, in the same place).

I have an existing solution in the tool I wrote to process the files:

Guid g = Guid.NewGuid();

string data = File.ReadAllText(file, Encoding.GetEncoding("Latin1"));
data = data.Replace("\r\n", g.ToString()); //just so I have a unique placeholder
data = data.Replace("\n", "");
data = data.Replace(g.ToString(), "\r\n");

However, this fails on files that are bigger than a gigabyte or so. (Also, I haven't profiled it, but I suspect it's dog slow as well).

The tools I have at my disposal are:

• cygwin tools (sed, grep, etc)
• .NET

What is the best way to do this?

Answer 1

Instead of reading the entire thing into memory as a big (potentially huge) string, consider a stream based approach instead.

Open the input stream and read a line at a time, making your replacements as needed. Open an output stream and write the modified line into it. Something like:

static void Main( string[] args )
{
    using( var inFs = File.OpenRead( @"C:\input.txt" ) )
    using( var reader = new StreamReader( inFs ) )
    using( var outFs = File.Create( @"C:\output.txt" ) )
    using( var writer = new StreamWriter( outFs ) )
    {
        int cur;
        char last = '0';
        while( ( cur = reader.Read() ) != -1 )
        {
            char next = (char)reader.Peek();
            char c = (char)cur;
            if( c != '\n' || last == '\r' )
                writer.Write( c );

            last = c;
        }
    }
}

Answer 2

That's an awful lot of code to do something so simple.

Try this instead.

tr -d '\n' <dirtyfile >cleanfile

Answer 3

Here's a StreamReader class that seems to do what I want. Note that this is probably incredibly domain specific, so it may or may not be useful:

class BadEOLStreamReader : StreamReader {
    private int pushback = -1;

    public BadEOLStreamReader(string file, Encoding encoding) : base(file, encoding) {

    }

    public override int Peek() {
        if (pushback != -1) {
            var r = pushback;
            pushback = -1;
            return r;
        }

        return base.Peek();
    }

    public override int Read() {
        if (pushback != -1) {
            var r = pushback;
            pushback = -1;
            return r;
        }

        skip:
        var ret = base.Read();
        if (ret == 13) {
            var ret2 = base.Read();
            if (ret2 == 10) {
                //it's good, push back the 10
                pushback = ret2;
                return ret;
            }
            pushback = ret2;
            //skip it
            goto skip;
        } else if (ret == 10) {
            //skip it
            goto skip;
        } else {

            return ret;
        }
    }
}

Answer 4

EDIT: after some tests, the awk solution gives better results in terms of speed.

The standard file/input filter in UNIX/Linux/Cygwin have a hard time dealing with binary file. To do that with filters, you need to convert your file in HEX, edit it with sed (or awk, see 2nd solution bellow), and convert it back to its original data. This should do it:

xxd -c1 -p file.txt | 
  sed -n -e '1{h}' -e '${x;G;p;d}' \
      -e '2,${x;G;/^0d\n0a$/{P;b};/\n0a$/{P;s/.*//;x;b};P}' |
  xxd -r -p

Ok, this is not simple to understand, let's begin with the simple parts:

• xxd -c1 -p file.txt converts file.txt from binary to HEX, one byte per line.
• xxd -r -p reverts the convertion.
• The sed replaces a \n (0a in HEX) that is not preceded by a \r (0d in HEX) by nothing.

The idea of the sed part is to store the previous byte in the hold space, and to deal with both the previous and the current byte:

• At the 1st line, store the line (byte) in the hold space.
• At the last line, print both bytes in the correct order (x;G;p) and stop the script (d).
• For the lines in between, after having the current byte in the hold space and the 2 bytes (previous and current) in the pattern space (x;G), 3 possibles cases:
  1. If is a \r\n, then print \r keeping \n in the hold space for the next cycle and stop this cycle (b command).
  2. Else if it ends with \n (meaning that it did not begin by \r) store an empty string in the hold space and stop this cycle (b command)
  3. Else print the 1st character.

It might be simpler to understand in awk:

xxd -c1 -p file.txt |
  awk 'NR > 1 && $0 == "0a" && p != "0d" {$0 = ""}
       NR > 1 {print p}
       {p = $0}
       END{print p}' |
  xxd -r -p

It can be tested with:

printf "foo,bar\nrab,baz\r\nalice,bob,chris\r\n" |
  xxd -c1 -p | 
  sed -n -e '1{h}' -e '${x;G;p;d}' \
      -e '2,${x;G;/^0d\n0a$/{P;b};/\n0a$/{P;s/.*//;x;b};P}' |
  xxd -r -p

or

printf "foo,bar\nrab,baz\r\nalice,bob,chris\r\n" |
  xxd -c1 -p |
  awk 'NR > 1 && $0 == "0a" && p != "0d" {$0 = ""}
       NR > 1 {print p}
       {p = $0}
       END{print p}' |
  xxd -r -p