0

For the first time, I am trying to work on a case study using python for continuous dataframe, which is the time series data of properties during the period 2006-2016

But I have missing values for the year 2015-16 in columns A,B,C,D and 2006-07 in E and F columns. I am trying to impute the values and fill the data.

**DataFrame**

I have tried MICE and Interpolation but am not sure if it's even correct or not. which method to apply and how to apply it in python? I have gone through links:

https://www.theanalysisfactor.com/seven-ways-to-make-up-data-common-methods-to-imputing-missing-data/ https://www.researchgate.net/post/What_is_a_reliable_method_of_dealing_with_missing_data_in_time_series_records

Should I be using forecasting method instead of imputation to fill the data?

Please help.

devCodePro
  • 97
  • 2
  • 4
  • 17

2 Answers2

2

In your case, if you fill your empty cells with estimated values, the results of your analysis will be very skewd. Because you have a very limited sample size.

If you have more data (e.g. more years), you can try different methods to fill the empty values in your dataset (interpolations, mean, etc). There are pros and cons for every method. It depends on what you need to do with this time series.

If you have only that data, it would make sense to use only the period in which you have the data for every column, but, again, having so few rows will led you to not so interesting results.

Anyway, pandas dataframes offer a lot of libs and utils to handle this problem.

For example the dataframe method fillna:

df = # your dataframe
df.fillna(method='ffill')

Which will propagate last valid observation forward to next valid

Or the interpolate method:

df.interpolate(method ='linear', limit_direction ='forward')

But there is no perfect answer to your question. You need to reason on your data and make a decision based on the context

Nikaido
  • 4,443
  • 5
  • 30
  • 47
  • Thanks for answering, Should I be using forecasting method instead of imputation to fill the data? – devCodePro Jan 31 '20 at 09:53
  • @PrakashShelke you could. There are different method for forecasting the data, but again, you don't have enough data I think. And question: how are you going to fill data in the past for the E and F columns? – Nikaido Jan 31 '20 at 11:23
  • I was thinking to use bfill but I am not sure if its the correct approach to have same values in multiple rows – devCodePro Jan 31 '20 at 11:46
  • Also I read about the MICE, but as the row is completly blank, the result is same. i.e. same values in past and future rows – devCodePro Jan 31 '20 at 11:53
2

There isn't always one best way to fill missing values in fact. Here are some methods used in python to fill values of time series.missing-values-in-time-series-in-python

Filling missing values a.k.a imputation is a well-studied topic in computer science and statistics.

Previously, we used to impute data with mean values regardless of data types. A big problem that mean imputation(all const imputation) triggers is mutations in time series.

Later, autoregressive(AR) and moving average(MA) used for modeling time series are used in imputation. These methods have a strong theoretical basis STAT510 and are used to forecast/impute time series.

Matrix Factorization is another important method, such as TRMF, SVD, PCA. A recent benchmark about MF imputation was published in PVLDB.Mind the Gap: An Experimental Evaluation of Imputation of Missing Values Techniques in Time Series.

Besides, there are other machine/deep learning methods proposed recently. There is a survey about imputation methods used in time seriesTime Series Data Imputation: A Survey on Deep Learning Approaches, which may help you a lot. However, the methods mentioned in this survey are not accurate enough.

Back to your question, MICE is just a framework where you can use any regression algorithms. It assumes that different columns(A, B, C, and E, F) are correlated.

Forecasting and imputation are the same by nature. You can think that forecasting is a special case of imputation without succeeding data.

You'd better try more imputation methods to find the best one.

LittleHealth
  • 104
  • 4