Export data efficiently to CSV using python

Question

I am using my arduino to analyze analog inputs and I am accessing the arduino using the pyfirmata library and I ambasically measuring voltages using the 6 analog inputs on my arduino Uno. I need to find a way to live time feed this data into a CSV efficiently... I am not sure on the best way to do that

Any suggestion would help but please write out the code you suggest. I would prefer to use Pandas if possible because it's easier

voltage0 through voltage5 are my variables and I am trying to report those in a nice format that will later have to be analyzed

import time
from datetime import datetime
import pyfirmata
import pandas as pd

board = pyfirmata.Arduino('/dev/ttyACM1')

analog_pin0 = board.get_pin('a:0:i')
analog_pin1 = board.get_pin('a:1:i')
analog_pin2 = board.get_pin('a:2:i')
analog_pin3 = board.get_pin('a:3:i')
analog_pin4 = board.get_pin('a:4:i')
analog_pin5 = board.get_pin('a:5:i')

it = pyfirmata.util.Iterator(board)
it.start()

analog_pin0.enable_reporting()
analog_pin1.enable_reporting()
analog_pin2.enable_reporting()
analog_pin3.enable_reporting()
analog_pin4.enable_reporting()
analog_pin5.enable_reporting()

data = []

count = 0
x = 0
start = 0

while x <= 1000:

reading0 = analog_pin0.read()
if reading0 != None:
    voltage0 = reading0 * 5
    voltage0 = round(voltage0,2)
else:
    voltage0 = float('nan')
reading1 = analog_pin1.read()    
if reading1 != None:
    voltage1 = reading1 * 5
    voltage1 = round(voltage1,2)
else:
    voltage1 = float('nan')
reading2 = analog_pin2.read()
if reading2 != None:
    voltage2 = reading2 * 5
    voltage2 = round(voltage2,2)
else:
    voltage2 = float('nan')
reading3 = analog_pin3.read()    
if reading3 != None:
    voltage3 = reading3 * 5
    voltage3 = round(voltage3,2)
else:
    voltage3 = float('nan')
reading4 = analog_pin4.read()
if reading4 != None:
    voltage4 = reading4 * 5
    voltage4 = round(voltage4,2)
else:
    voltage4 = float('nan')
reading5 = analog_pin5.read()    
if reading5 != None:
    voltage5 = reading5 * 5
    voltage5 = round(voltage5,2)
else:
    voltage5 = float('nan')

datarow = {'Voltage0': voltage0, 'Voltage1': voltage1, 'Voltage2' : voltage2, 'Voltage3': voltage3, 'Voltage4' : voltage4, 'Voltage5' : voltage5, 'Time' : time.strftime("%Y-%m-%d_%H:%M:%S")}
data.append(datarow)

if count%500 == 0:
    dataframe = pd.DataFrame(data)
    dataframe.to_csv('data.csv')

x += 1
count += 1

#time.sleep(1)enter code here

using chunk sizing will help. [chunk_size](https://stackoverflow.com/questions/38531195/writing-panda-dataframes-to-csv-file-in-chunks) — Axois, Jul 17 '19 at 01:10

score 0 · Answer 1 · answered Jul 17 '19 at 01:04

Your code seems to work, but it's not very efficient. Every 500 iterations, you rewrite all your data instead of updating your file with the new data in the end. You might consider saving it this way instead:

if count%500 == 0:
    dataframe = pd.DataFrame(data)
    dataframe.to_csv('data.csv',mode='a',header=False)
    data = []

If it's still not fast enough, you might consider saving your data to a binary format such as .npy (numpy format), and convert it later to csv.

Export data efficiently to CSV using python

1 Answers1