Remove entire row from CSV (if blank)

Question

I have a data file from an instrument that outputs as a CSV. Reading the file and the corresponding columns are no issue, however, due to a slight change in instrumentation, the data file has changed and I'm not sure how to change my code to still read the file.

f = open('Rotator_050816.dat')
lines = f.readlines()
i = 0
while (lines[i]<>"[Data]\n"):
   i+=1
i = i + 2
Temp = []; Field = []; Resistance1 = []; Resistance2 = [];
while(i<len(lines)):
    data = lines[i].split(",")
    Temp.append(float(data[3])
    Field.append(float(data[4])
    Resistance1.append(float[12])
    Resistance2.append(float[13])
i+=1

Temp = np.array(Temp)
Field_T = np.array(Field)/10000.
Resistance1 = np.array(Resistance1)
Excitation1 = np.array(Excitation1)

This is a MWE from previous usage. This has no issue if the CSV file has no blank entries, however, if there are blank entries it presents a problem as then len(Resistance1) ≠ len(Temp) so they cannot be plotted correctly. So my data file now looks like this:

Example Data File

So I need to add lines of code that can read if a row for Res. Ch1 or Res. Ch2 is empty, and then skip that entire row for all variables before appending to the final set of data. This way len(Resistance1) = len(Temp) and each Res. Ch1 measurement matches up to the right Temperature.

This needs a [MCVE] (MWE). What is `[data]`? Also, `<>` isn't valid Python syntax. Plus, that while loop will never terminate. — Morgan Thrapp, Jun 23 '16 at 18:17
Added more to the MWE. It is fully working for data files with no empty spaces. — , Jun 23 '16 at 18:37

touch my body · Answer 1 · 2016-06-23T18:25:40.397

1) Open the file in read-only mode and get all the lines

lines_in_my_file = []
with open("my_file.csv", "r") as my_file:
    lines_in_my_file = my_file.readlines()

2) Open the file again, this time in write mode, and write all non-blank lines into the file:

with open("my_file.csv", "r") as my_file:
    for line in lines_in_my_file:
        if line.strip().strip(",") != ""
            my_file.write(line)

Keep in mind, this will remove any line that's made up of just spaces, tabs, or commas. So any rows that look like these:

,,,, (this line has only commas)
     (this line has only spaces)
\n   (this line is just a newline character)

...will be deleted.

score 1 · Answer 2 · answered Jun 24 '16 at 01:50

Here is my working solution that I have implemented:

while (i<len(lines)):
data = lines[i].split(",")
    if float(data[4]) >30000 and float(data[4]) <50000:
        Temp_II.append(float(data[3]))  #As K
        Field_II.append(float(data[4]))   #As Oe
        Position_II.append(float(data[5])) #As Degree
        #loop for Resistivity1 column cleanup
        if data[12]!= '':
            Resistivity1_II.append(float(data[12]))
            Temp1_II.append(float(data[3]))
        #loop for Resistivity2 column cleanup
        if data[13]!= '':
           Resistivity2_II.append(float(data[13]))
           Temp2_II.append(float(data[3]))                        
i+=1

Basically, this pairs up the Resistivity1 entries that are not blank with the corresponding Temperature entries and the same for Resistivity2.

Remove entire row from CSV (if blank)

2 Answers2