Parsing two files with Python

Question

I'm still new to python and cannot achieve to make what i'm looking for. I'm using Python 3.7.0

I have one file, called log.csv, containing a log of CANbus messages.

I want to check what is the content of column label Data2 and Data3 when the ID is 348 in column label ID.

If they are both different from "00", I want to make a new string called fault_code with the "Data3+Data2".

Then I want to check on another CSV file where this code string appear, and print the column 6 of this row (label description). But this last part I want to do it only one time per fault_code.

Here is my code:

import csv

CAN_ID = "348"

with open('0.csv') as log:
    reader = csv.reader(log,delimiter=',')
    for log_row in reader:
        if log_row[1] == CAN_ID:
            if (log_row[5]+log_row[4]) != "0000":
                fault_code = log_row[5]+log_row[4]
                with open('Fault_codes.csv') as fault:
                    readerFC = csv.reader(fault,delimiter=';')
                    for fault_row in readerFC:
                        if "0x"+fault_code in readerFC:
                            print("{fault_row[6]}")

Here is a part of the log.csv file

Timestamp,ID,Data0,Data1,Data2,Data3,Data4,Data5,Data6,Data7,                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
396774,313,0F,00,28,0A,00,00,C2,FF
396774,314,00,00,06,02,10,00,D8,00
396775,**348**,2C,00,**00,00**,FF,7F,E6,02

and this is a part of faultcode.csv

Level;LED Flashes;UID;FID;Type;Display;Message;Description;RecommendedAction 
1;2;1;**0x4481**;Warning;F12001;Handbrake Fault;Handbrake is active;Release handbrake 
1;5;1;**0x4541**;Warning;F15001;Fan Fault;blablabla;blablalba 
1;5;2;**0x4542**;Warning;F15002;blablabla

Also do you think of a better way to do this task? I've read that Pandas can be very good for large files. As log.csv can have 100'000+ row, it's maybe a better idea to use it. What do you think?

Thank you for your help!

You can nest `with` blocks as much as you want. However, in Python 3 you cannot mix tabs and spaces in your indentation. — PM 2Ring, Sep 18 '18 at 12:47
BTW, opening the same file over & over in a loop (like 'Fault_codes.csv') isn't very efficient. It's better to open it outside the loop, and either read the whole thing into memory, or use the `.seek` method to "rewind" it if you really need to read it multiple times. — PM 2Ring, Sep 18 '18 at 12:49
Simple Googling gave me this. Inconsistency is in the usage of space and tabs.https://stackoverflow.com/questions/5685406/inconsistent-use-of-tabs-and-spaces-in-indentation — mad_, Sep 18 '18 at 12:50
Possible duplicate of ["inconsistent use of tabs and spaces in indentation"](https://stackoverflow.com/questions/5685406/inconsistent-use-of-tabs-and-spaces-in-indentation) — mad_, Sep 18 '18 at 12:50
Okay I solved the problem of indentation by using only tab and set on my editor the correct length of tabulation. Thank you for pointing this out. — TheClerc, Sep 18 '18 at 13:10

score 0 · Accepted Answer · answered Sep 18 '18 at 12:58

Be careful with your indentation, you get this error because you sometimes you use spaces and other tabs to indent.

As PM 2Ring said, reading 'Fault_codes.csv' everytime you read 1 line of your log is really not efficient.

You should read faultcode once and store the content in RAM (if it fits). You can use pandas to do it, and store the content into a DataFrame. I would do that before reading your logs.

You do not need to store all log.csv lines in RAM. So I'd keep reading it line by line with csv module, do my stuff, write to a new file, and read the next line. No need to use pandas here as it will fill your RAM for nothing.

Alright! I'll check how pandas work with DataFrame and then compare with the other file on the loop. — TheClerc, Sep 18 '18 at 13:13

Parsing two files with Python

1 Answers1