5

I have a CSV file that has data from a random sensor recorded over a few minutes time. Now I want to stream that data from a CSV file to my python code as if it were receiving data from the sensor itself directly. (The code is for taking readings from two different sensors/csv files and averaging them.) Someone suggested to use Apache Spark to stream data, but I feel that's a bit too complex for me. Might there be a simpler solution?

Patrick Yoder
  • 1,065
  • 4
  • 14
  • 19
M. Ali
  • 51
  • 1
  • 1
  • 2
  • 1
    Can't you just read it line by line? (and optional add a bit of delay between reading a line) – Olian04 Jan 17 '17 at 18:43
  • 1
    Are you wanting to mock the sensor interface or do you just care about the data? Do you need to have delays between reads? Is the data timestamped so you can calculate delays if you need them? – Steven Rumbalski Jan 17 '17 at 18:44
  • @Olian04 perhaps i could do that. i am pretty new to programming and have no idea about all this stuff. i apologize for sounding stupid. but will i have to give some time delay in between reading the lines to give it the real effect? – M. Ali Jan 18 '17 at 15:23
  • @M.Ali not necessarily. If the time between reading doesn't matter then you can just read it as you would any other file. – Olian04 Jan 19 '17 at 08:44
  • There is an answer in https://stackoverflow.com/a/6556862/1548275 – Jari Turkia Nov 12 '21 at 10:46

4 Answers4

6

You could also use pandas read_csv() function to read the big csv file in small chunks, the basic code is written below:

import pandas as pd
chunksize = 100
for chunk in pd.read_csv('myfile.csv', chunksize=chunksize):
    print(chunk)

This link explains how this works: http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking

Shashank Srivastava
  • 195
  • 1
  • 12
0

you could use something like tail -f in python to achieve this. this should do what you want. http://lethain.com/tailing-in-python/

acushner
  • 9,595
  • 1
  • 34
  • 34
  • is it possible if I make a virtual serial device and transmit the data through that? i mean ultimately thats how i will interface the sensors anyway, so why not code according to that? – M. Ali Jan 18 '17 at 17:00
  • no idea, but probably? seems more complicated than just reading the csv files, unless you want to flip it around and have python interface directly with the sensors and have the python process also store csv data. – acushner Jan 18 '17 at 22:42
0

You can also work with Python on Numpy/Matplotlib. It´s a easy way to stream your csv data temporay as a variable and not a extra file. 

´import matplotlib.pyplot as plt
 from matplotlib import style 
 import numpy as np
 import io

def draw_graph_stream(csv_content):

csv_stream = io.StringIO(csv_content)
svg_stream = io.StringIO()

data =  np.genfromtxt(csv_stream, delimiter = ';')          # generate the stream

x = data[0,:]                           #first row in csv 
y = np.mean(data[1:,:], axis=0)  # first column with mean generate the average           

plt.plot(x,y)
plt.savefig(svg_stream, format = 'svg')     #just safe it as svg 

svg_stream.seek(0)           #Position 0 for reading after writing 
return svg_stream.read()

print("Start test")

with io.open('/filepathtodata','r') as csv_file:   #works like a Loop 
    print("Reading file")
    csv_content = csv_file.read()
    print("Drawing graph")
    svg_content = draw_graph_stream(csv_content)

    with io.open('thefilepathforsafe','w+') as svg_file:
        print("Write back")
        svg_file.write(svg_content)´
Tobi
  • 1
  • 1
0

It had worked in fastapi for streaming data to a user interface.

from starlette.responses import StreamingResponse
from io import BytesIO

temp_file_name = 'test.csv'
with open(temp_file_name, 'rb') as fh:
    buffered_data = BytesIO(fh.read())
    response = StreamingResponse(buffered_data, media_type="text/csv")
    response.headers["Content-Disposition"] = f"attachment; filename=filename.csv"
    return response
Rakesh Dara
  • 27
  • 4