How to read from CSV file

Question

I am trying to understand how Kalman Filter for non-linear system works. While searching for an example, I cam across this good basic example.

import numpy as np
import pylab as pl
import pandas as pd
from pykalman import UnscentedKalmanFilter

# initialize parameters
def transition_function(state, noise):
    a = np.sin(state[0]) + state[1] * noise[0]
    b = state[1] + noise[1]
    return np.array([a, b])

def observation_function(state, noise):
    C = np.array([[-1, 0.5], [0.2, 0.1]])
    return np.dot(C, state) + noise

transition_covariance = np.eye(2)
random_state = np.random.RandomState(0)
observation_covariance = np.eye(2) + random_state.randn(2, 2) * 0.1
initial_state_mean = [0, 0]
initial_state_covariance = [[1, 0.1], [-0.1, 1]]

# sample from model
kf = UnscentedKalmanFilter(
    transition_function, observation_function,
    transition_covariance, observation_covariance,
    initial_state_mean, initial_state_covariance,
    random_state=random_state
)
states, observations = kf.sample(50, initial_state_mean)

# estimate state with filtering and smoothing
filtered_state_estimates = kf.filter(observations)[0]
smoothed_state_estimates = kf.smooth(observations)[0]

# draw estimates
pl.figure()
lines_true = pl.plot(states, color='b')
lines_filt = pl.plot(filtered_state_estimates, color='r', ls='-')
lines_smooth = pl.plot(smoothed_state_estimates, color='g', ls='-.')
pl.legend((lines_true[0], lines_filt[0], lines_smooth[0]),
          ('true', 'filt', 'smooth'),
          loc='lower left'
)
pl.show()

This code produces the following graph.

However,for my experiment - I have created a very small time series data ready with three columns formatted as follows. The full dataset is attached here for reproduciability.

  time        X      Y
 0.040662  1.041667  1
 0.139757  1.760417  2
 0.144357  1.190104  1
 0.145341  1.047526  1
 0.145401  1.011882  1
 0.148465  1.002970  1
 ....      .....     .

Instead of using the random values as shown in the code, how can we input from the CSV file I attached? Here is my approach, but it doesn't seem to workout for me and I would appreciate for any help.

df = pd.read_csv('testdata.csv')
pd.set_option('use_inf_as_null', True)

df.dropna(inplace=True)

X = df.drop('Y', axis=1)
y = df['Y']


d1= np.array(X)
d2 = np.array(y)

I think you'll find your answer here: https://stackoverflow.com/questions/43749472/how-to-save-certain-columns-from-csv-file-as-variables-in-python — Rankinstudio, Dec 29 '18 at 17:10
@AI_Learning, this is the error i get `ValueError: could not broadcast input array from shape (377,2) into shape (377)` — , Dec 30 '18 at 18:29
Please post the complete traceback of the error else it's difficult to understand where you got this error. — Venkatachalam, Dec 31 '18 at 01:44
@AI_Learning, What I did is - you see the `return np.array([a, b])` in the first function? I replaced `a` and `b` with `X` and `y` in my approach as you can see at the end of my post. Then finally, I did `return np.array([d1, d2])`. — , Dec 31 '18 at 10:52

score 0 · Answer 1 · answered Dec 29 '18 at 19:41

0

From the link I shared, here is how you get the CSV data into Numpy Arrays.

import numpy as np
import csv

with open('testdata.csv','r') as csvfile:
    r = csv.reader(csvfile, delimiter=',')
    data = [i for i in r]

headings = data.pop(0)
data = np.array([[np.float(j) for j in i] for i in data])

T = data.T[0] #Time
X = data.T[1] #X
Y = data.T[2] #Y

print(T)
print(X)
print(Y)

answered Dec 29 '18 at 19:41

Rankinstudio

591
6
19

I would therefore appreciate if you could try to reproduce what I posted and update your answer with a working solution so that I mark your answer as an accepted. Thank you. – Dec 30 '18 at 11:06
"Instead of using the random values as shown in the code, how can we input from the CSV file I attached?" <- that was the question you asked. I showed you how to import the CSV values into numpy. – Rankinstudio Dec 30 '18 at 19:42

How to read from CSV file

1 Answers1