Compare lines in 2 text files

Question

I have two large text files (200,000+ lines), CSV format. I need to compare them, line by line, but the fields maybe switched within each line.

Example file A.csv:

AAA,BBB,,DDD  
EEE,,GGG,HHH  
III,JJJ,KKK,LLL

Example file B.csv:

AAA,,BBB,DDD  
EEE,,GGG,HHH  
LLL,KKK,JJJ,III

So for my purposes, A.csv and B.csv should be "identical" even though fields are switch in the first and last line. Since the fields in each file might be in a different order, the usual options like grep or diff won't work.

Basically, I think I need to write something that reads a line of A.csv and B.csv, and checks if all fields are present in both lines, independent of the order. Alternatively, something that orders the fields after reading the lines.

Burhan Khalid · Accepted Answer · 2016-01-10T07:01:07.427

6

You can normalize the check, without affecting the data.

with open('big1.csv') as i, open('big2.csv') as j:
   a = csv.reader(i)
   b = csv.reader(j)
   for linea in a:
      lineb = next(b)
      if sorted(map(str.lower, linea)) != sorted(map(str.lower, lineb)):
          print('{} does not match {}'.format(linea, lineb))

edited Jan 10 '16 at 07:01

answered Jan 10 '16 at 06:04

Burhan Khalid

169,990
18
245
284

You can replace lines 4 and 5 with `for linea, lineb in zip(a, b):` – fenceop Jan 10 '16 at 07:01
You can, but this would load all the file in memory and the question is specifically asking for large files. – Burhan Khalid Jan 10 '16 at 07:02
Even if `zip()` creates an iterator? – fenceop Jan 10 '16 at 07:09
@fenceop Only on Python 3.x; and since its not tagged specifically - the solution with manual iteration will work in both versions. – Burhan Khalid Jan 10 '16 at 07:25
@BurhanKhalid...One question and forgive lack of knowledge here...but does reading a line in a csv file skips the emtpy fields, like in `AAA,BBB,,DDD` ? so the linea will be `['aaa','bbb',ddd']`? – Iron Fist Jan 10 '16 at 08:38
No - you'll get `['AAA','BBB','','DDD']` for `'AAA,BBB,,DDD'`. If you want to ignore blanks in your comparison, you'll have to filter them out. – Burhan Khalid Jan 10 '16 at 09:16
@BurhanKhalid Tested on minimal example files, works like a treat! Thank you! – Markus Heller Jan 11 '16 at 19:06
@BurhanKhalid How can I filter out the empty fields? – Markus Heller Jan 12 '16 at 20:34

score -1 · Answer 2 · edited Jan 10 '16 at 07:00

-1

Try using diff as a Linux/Unix command line - this is very useful for comparing files.

edited Jan 10 '16 at 07:00

Falko

17,076
13
60
105

answered Jan 10 '16 at 06:36

Tim Seed

5,119
2
30
26

Compare lines in 2 text files

2 Answers2

Linked