How to check the fieldname before to write all values in csv file

Question

I want to write a dictionary in csv file. With for loop i write the fieldnames "keys" "i mean tittles of columns" in csv file

Desired is: before to write the values I want to control if values belong to the right filed in the csv file. I tried something like this but when I change the order of dict. I get the values in the wrong fields

lineValue = ""
for k,v in data.items():
    if k[]=v[]:
        lineValue+= v + ","
lineValue = Time + "," + lineValue[:-1] + "\n" 
outfile = open(filename, "a")
outfile.write(lineValue)
outfile.close()

I think your lineValue building is strange ... could you give a sample of what you have before and what you expect ? to me, your' if' condition is wrong as you do an assignment instead of a comparison ... — A.Joly, Oct 31 '17 at 16:42
lineValue is a string with all values which i got from a dictionary data{} in this question you can seee the whole code how i programme it. [link] (https://stackoverflow.com/questions/47034977/how-control-the-place-of-value-with-the-place-of-key-of-dictionary-before-writin?noredirect=1#comment81017207_47034977) — AhmyOhlin, Oct 31 '17 at 16:48
what i expect is: i want to control the fieldname before to write the value in each column. So i will be 100% sure that the value belongs to the right key. I hope that you understand what i want to do — AhmyOhlin, Oct 31 '17 at 16:51
so you want to write 'Time, value' (from your code), and you want to check that the value corresponds to the right key ? can you write a sample of what you expect in your csv file, from which input, it's not clear to me ... — A.Joly, Oct 31 '17 at 16:56
yes exactly. Time, ValueSensor1,ValueSensor2,ValueSensor3... The problem is when one of the sensors is shut down, then i lose one value in LineValue where i store all the Sensorvalues. All the values will be written in wrong keys. — AhmyOhlin, Oct 31 '17 at 17:08
does your key contains the information about the sensor concerned ? ie. key=sensor1, etc. ? If so, you could build a list with the expected keys and loop over these values in parallel. And if the key doesn't match an expected one, fill value with 0 or another neutral value ... — A.Joly, Nov 03 '17 at 07:38
Yes exactly that is what i have for eg. TemparaturSensor= 20 °C. i stored values of alles sensors in dictionary with {Sensortype: Value}. After that i write sensor name as titles of my csv file for one time and after that i write only the values everytime in csv fine. Could you give me please an example how i can do that. I have neve do that before. it would be helpfull when you can give me an example with. "If so, you could build a list with the expected keys and loop over these values in parallel" — AhmyOhlin, Nov 03 '17 at 13:56

score 0 · Answer 1 · answered Oct 31 '17 at 16:29

0

You can try something like this:

import csv

ofile  = open('ttest.csv', "wb")
writer = csv.writer(ofile, delimiter='', quotechar='"', quoting=csv.QUOTE_ALL)

for row in reader:
    writer.writerow(row)

ofile.close()

answered Oct 31 '17 at 16:29

Bzzoiro

89
4

how can wou check the the write fieldname with your code with for row in reader: you get the entire of row – AhmyOhlin Oct 31 '17 at 16:36

score 0 · Answer 2 · edited May 05 '23 at 08:03

Let's say you have 3 sensors name sensor1, sensor2, sensor3 in your system. Assume it is a well-know configuration, you can build a list of expected sensors to check:

mySensors=['sensor1', 'sensor2', 'sensor3']

Then you have a dictionary data in which you store sensor data that you read, ie.

data={'sensor1':value1;'sensor2':value2;'sensor3':value3}

Sometimes you have missing value if, let's say sensor2 is OFF, then you'll end up with

data={'sensor1':value1;'sensor3':value3}

is that right ?

then with your current code, you miss the value and write sensor3's value in sensor2 column of the CSV.

To avoid this you may loop over your known configuration, ie. the list. note that this is valid only if you have a well-known and fixed sensor configuration

with open(filename,'a') as outfile:
    lineValue = ""
    for component in mySensors:
        # check that the sensor is in the data read
        if component not in data:
            # the sensor has not been read, set a neutral value
            lineValue+= 'n/a' + ","
        else:
            v = data[component]
            lineValue+= str(v) + ","  # I cast value for my test, your value is probably already in correct format
        #for k,v in data.items():
        #    if k[]=v[]: => Can you explain that comparison, please ? 
        #        lineValue+= v + ","
    lineValue = Time + "," + lineValue[:-1] + "\n" 
    outfile.write(lineValue)

This should handle value shifting. You can set any neutral value instead of 'n/a'

If data is in the second configuration (ie, missing sensor2), the line you'll write in your file will be:

'Time,5,n/a,20\n'

EDIT: upgrade from your comments

So I understand that you may have missing values from known sensors (already in CSV file), new values for known sensor and new values from new sensors.

First get your header line in CSV (which should look like 'duration, sensor1, sensor2, ... sensorx')

with open(filename,'r') as myFile:
    readline = ""
    # parse file until you hit the header line
    for line in myFile:
        readline = myFile.read()
        if readline != "":
            break

From this, get the list:

mySensors = readline.split(',')
# should give ['duration','sensor1','sensor2', etc.] => remove 'duration' column header

Then parse this sensorList. So if you hit a sensor from the list without value, it will write 'n/a'

To add new sensors that may have appear between two CSV file update, add a second 'for' loop after the first one:

# then check if there are new sensors that appeared and that are not in current set of sensors
header_update = False # to check if header has to be updated

for sensor, value in data.items():
    if sensor not in mySensors:
        # new sensor to add 
        lineValue+=str(value)
        # add the new sensor id in the header list
        mySensors.append(sensor)
        if not header_update : header_update = True
        
# terminate the line to write
lineValue = Time + "," + lineValue[:-1] + "\n"

After that, the problem is to add the new sensor to your header line, so you have to build the new header line

# new header line to replace the previous header
if header_update:
    new_header = "duration," # assuming that your first column header is 'duration'
    new_header += ','_join.mySensors
# this will build a new header line like duration,sensor1,sensor2,..., sensorx, newsensor1, etc.

and to replace it in the file.

To do so you can have a look to this post (I did test the answer of Kiran for my own purpose and it works pretty well, just make a backup of your CSV file until your script works properly to avoid unwanted file destruction)

thank you a lot for your pretty detailed ansswer. the Thing is: I dont have a fixed list of mySensors. I get the list of my sensors automatically "for example a new sensor4 will be directly detected and added to a list" which means when the sensor2 is missed at 15.00, its value is missed as well. But the CSV file which is created everyday at 00:00 with all keys "SensorsNames" still has the name of the missed Sensor2. This why i think — AhmyOhlin, Nov 03 '17 at 15:32
i can not check the missed Sensor2 from the actual data with: for component in mySensors: # check that the sensor is in the data read if component not in data: What i want is to open the created CSV file and read the titles of my sensors and write it in a list and then compare it with values. What do you think? — AhmyOhlin, Nov 03 '17 at 15:32
Sorry for my late answer. i just fixed some techncal issues and now iam able to work on Code again. the first function to get header line give me a stored values but not the header line with open(filename,'r') as myFile: readline = "" — AhmyOhlin, Nov 21 '17 at 09:57
how can i get only the header line with namefields of all my sensors. I tried with readline = myFile.read([0]) but i doesnt work — AhmyOhlin, Nov 21 '17 at 10:03
do you actually have a header line ? how do the first two lines of your file look like ? — A.Joly, Nov 21 '17 at 15:26
i got the header with this part of code: with open(filename,'r') as myFile: reader = csv.DictReader(myFile) fieldnames = reader.fieldnames for i in fieldnames: indexFieldnames = fieldnames.index(i) print("") print("index of fieldnames is:", indexFieldnames, "and name of field is: ", i) myFile.close() — AhmyOhlin, Nov 21 '17 at 15:30
ok, that's fine then. So you can go through the whole process ? does it work ? — A.Joly, Nov 21 '17 at 15:57
No. It doesnt work. I provide all the script and the input in a new question. could you take a look and to understand my problem exactly. https://stackoverflow.com/questions/47412324/compare-values-with-fieldnames-before-writing-it-in-csv-file-using-python — AhmyOhlin, Nov 21 '17 at 16:00

How to check the fieldname before to write all values in csv file

2 Answers2