0

I am having to add couple of lists in python as columns to an existing CSV file. I want to make use of a temporary file for the output CSV because I want to sort first 2 columns of that resulting data and then write to a new final CSV file. I don't want to keep the unsorted csv file which is why I am trying to use tempfile.NamedTemporaryFile for that step. It's giving nothing in the final CSV file but no other code errors. I changed how the with blocks are indented but unable to fix it. I tested by using a file on disk which works fine. I need help understanding what I am doing wrong. Here is my code:


# Open the existing csv in read mode and new temporary csv in write mode
with open(csvfile.name, 'r') as read_f, \
    tempfile.NamedTemporaryFile(suffix='.csv', prefix=('inter'), mode='w', delete=False) as write_f:
    csv_reader = csv.reader(read_f)
    csv_writer = csv.writer(write_f)
    i = 0
    for row in csv_reader:
        # Append the new list values to that row/list 
        row.append(company_list[i])
        row.append(highest_percentage[i])
        # Add the updated row / list to the output file
        csv_writer.writerow(row)
        i += 1

    with open(write_f.name) as data:
        stuff = csv.reader(data)  
        sortedlist = sorted(stuff, key=operator.itemgetter(0, 1))
    #now write the sorted result into final CSV file
    with open(fileout, 'w', newline='') as f:
        fileWriter = csv.writer(f)
        for row in sortedlist:
            fileWriter.writerow(row)
dna-data
  • 73
  • 5

2 Answers2

0

You should insert a write_f.seek(0, 0)

Just before the line opening the temporary file:

    write_f.seek(0, 0)
    with open(write_f.name) as data:
nilleb
  • 967
  • 6
  • 28
  • Ah just did that... getting this error now. IndexError: list index out of range – dna-data May 19 '20 at 19:50
  • Would you have a traceback? I have tested your code in repl.it, and it kinda works – nilleb May 19 '20 at 19:51
  • Given the error you get, I would say that one of the arrays company_list or highest_percentage have less values than rows in your file – nilleb May 19 '20 at 19:52
  • Thanks for feedback @nilleb! It does look like list items don't match. But I am positive they do because if I replace temp file with a local one, it works as expected. This is the result from adding the f.seek : Traceback (most recent call last): File ".\src\myprogram.py", line 148, in sortedlist = sorted(stuff, key=operator.itemgetter(0, 1)) IndexError: list index out of range – dna-data May 19 '20 at 19:55
  • So, one of the lines in the CSV is a list of fields of length 1 (ie. `line[0]` works well, but `line[1]` raises a `IndexError`) – nilleb May 19 '20 at 20:16
  • That suggests my code has issues while writing to that temp file. I was wondering if my problem was around reading from it but it may be before that. Another option( to avoid this method ) is for me to delete the intermediate csv file – dna-data May 19 '20 at 20:25
  • May I ask you to mark this as the answer to your question, since the question was about the content of the temporary CSV file, please? – nilleb May 19 '20 at 20:37
0

I found out what was causing the IndexError and consequently the empty final CSV. I resolved it with the help of this: CSV file written with Python has blank lines between each row. Here's my changed code that worked as desired: 

with open(csvfile.name, 'r') as read_f, \
    tempfile.NamedTemporaryFile(suffix='.csv', prefix=('inter'), newline='', mode='w+', delete=False) as write_f:
    csv_reader = csv.reader(read_f)
    csv_writer = csv.writer(write_f)
    i = 0
    for row in csv_reader:
        # Append the new list values to that row/list 
        row.append(company_list[i])
        row.append(highest_percentage[i])
        # Add the updated row / list to the output file
        csv_writer.writerow(row)
        i += 1

with open(write_f.name) as read_stuff, \
    open(fileout, 'w', newline='') as write_stuff:
    read_data = csv.reader(read_stuff)
    write_data = csv.writer(write_stuff)
    sortedlist = sorted(read_data, key=operator.itemgetter(0, 1))
    for row in sortedlist:
        write_data.writerow(row)
dna-data
  • 73
  • 5