1

This is probably a basic question, so my apologies if it has been asked before, I have searched extensively and not been able to find an answer.

I am reading records in protobuf format, and trying to come up with a script that will write to csv. The proto file has lots of optional messages, followed by a value. I want to be able to write the value to a corresponding column.

Eg

           A , B , C , D , E , F , G , H

columns

The proto messages will be a stream of random values matched to a column heading.

ie (A,1) (B,4), (H,2), (F,3)

(much more complicated, but this will do for an example). When i receive a message, i want to be able to locate the correct column, and place the value directly into it.

Note: I am writing this for others to use, so would prefer not to use Panda's for simplicity's sake. There are hundreds of columns, so is there anyway to place the value directly into a column without searching through all columns each time using == to find the corresponding one? Ie something along the lines of :

write value 3 to column F

Geordie Wicks
  • 1,065
  • 1
  • 11
  • 27
  • Do you need the file to always be up to date or can you write to it once, when the stream is closed? How do you control switching to the next row? How do you handle missing values? – Eugene Pakhomov Feb 25 '20 at 09:06
  • No I am writing after the stream is closed. Each proto message has a repeated field, so for each message I write all the above into the columns, then when I switch to the next message i just write a \n – Geordie Wicks Feb 25 '20 at 09:12

1 Answers1

0

You could turn your CSV into a dict, see Creating a dictionary from a csv file?

Then replace your dict values based on the column name / key.

Then rewrite the whole thing at once. If you need to preserve order just keep the original header row.

Cedric Druck
  • 1,032
  • 7
  • 20
  • Each row needs maintain consistency. So row 1 will have 10 values out of a possible 200, row 2 will have 25 values, row 3 47 values etc. So the final csv will have a lot of blank spaces. I guess i could fill with a value like -1, but that would be somewhat expensive – Geordie Wicks Feb 25 '20 at 09:09
  • @GeordieWicks https://docs.python.org/3/library/csv.html#csv.DictWriter will handle the missing values for you. You just have to make sure you know the set of all fields in advance. – Eugene Pakhomov Feb 25 '20 at 11:02
  • Thanks @EugenePakhomov, that was the library I was looking for :) – Geordie Wicks Feb 26 '20 at 12:20