Difference and Merging 2 Text Files

Question

So I have a time limited problem (the subroutine should be as fast as possible) for doing this. At any point of file I have 2 network files, like this:

Time: 23.369
1464953512
Bytes: 4193304
Time: 24.369
1464953502
Bytes: 4194304

Time: 24.369
1464953502
Bytes: 4194304
Time: 25.404
1464953527
Bytes: 4194304

Take note the 2 files are different and each contain a subset of information, and I need to merge these 2 to create the cumulative information(removing duplicates), like this:

Time: 23.369
1464953512
Bytes: 4193304
Time: 24.369
1464953502
Bytes: 4194304
Time: 25.404
1464953527
Bytes: 4194304

What is the fastest way to do this? (with some code please, if possible).

Thanks.

P.S. I was looking at some diff/merge libraries but I think it will be a overkill for the same. Any simple .net/LINQ magic that can achieve it? Also, the duplicates are serial, as shown and not scattered around.

EDIT: -ve voters please leave a comment so that I can improve or otherwise change the question to be more suitable.

Are those numbers `1464953512` a key of some kind? Does it belong to the Bytes/Time below or above? If so, you can just smash objects from both files into a set and use the set to write your final file. Alternatively, LINQ's except method is what you're looking for. — saarrrr, Jun 07 '16 at 22:29
@saarrrr No the numbers are totally random timestamps from a rpc. They are useless and are just there for some legacy/other reasons maybe. — Jishan, Jun 07 '16 at 22:32

score 0 · Accepted Answer · answered Jun 10 '16 at 19:20

If this helps anybody, I used this to find the union of 2 text files, having converted them into string enum before:

var dinfo = new DirectoryInfo(@"C:\http");
var files = dinfo.GetFiles("*.txt");
IEnumerable<string> _eValA = null;
IEnumerable<string> _eValB = null;

_eValA = File.ReadLines(@"C:\http\http1.txt");
_eValB = File.ReadLines(@"C:\http2.txt");

IEnumerable<String> union = _eValA.Union(_eValB);

//TODO: create file if does not exist
File.WriteAllLines(@"C:\http\union.txt", union.Cast<String>());

score 0 · Answer 2 · answered Jun 10 '16 at 19:36

You'll want to read in the files, creating instances of a custom class as you go with two properties: Time and Bytes. In your custom class, override the Equals and GetHashCode methods and have them use the Time property. For example:

public override int GetHashCode() {
    return Time.GetHashCode();
}

public override bool Equals(obj other) {
    //skipping type check and null check for brevity
    return Time.Equals(other.Time);
}

Then just add your items to a HashSet<YourCustomClass>. HashSet doesn't allow duplicates, so you'll be good to go.

Difference and Merging 2 Text Files

2 Answers2