Python-Predicting/Extrapolating future data given a data set

Question

I'm very new to Python. I have a data set and I'm trying to use numPy/sciPy to predict/extrapolate future data points. Is there a simple way to come up with a mathematical function(say, a Sine function) that fits my current data, and then I could pass new values into that function to get my prediction?

Here's what I have but I don't think it's doing what I want:

import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

def main():

    y = [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
         12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
         7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
         11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
         10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
         11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
         12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
         8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
         11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
         8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
         12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
         8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
         8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
         11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
         9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
         13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3]    
    x = np.array(np.arange(len(y)))        

    fitting_parameters, covariance = curve_fit(fit, x, y)
    a = fitting_parameters[0]
    b = fitting_parameters[1]
    c = fitting_parameters[2]
    d = fitting_parameters[3]

    for x_predict in range(len(y) + 1, len(y) + 24):
        next_x = x_predict
        next_y = fit(next_x, a, b, c, d)

        print("next_x: " + str(next_x))
        print("next_y: " + str(next_y))
        y.append(next_y)

    plt.plot(y)
    plt.show()

def fit(x, a, b, c, d):
    return a*np.sin(b*x + c) + d

I tried to curve_fit and univariatespline my data but that only fits my current data and smooths my points, respectively. My point is that these tools are only "fitting" my data but aren't actually giving me a function that I can use to get future points.

I thought I could use Discrete Fourier Transforms because my data is periodic and looked like it could be described as a summation of Sines and Cosines. But once I got a frequency domain from the time domain, I was stuck on how to "extrapolate" in order to predict future periods and points in the time domain:

import numpy as np
import matplotlib.pyplot as plt

mydata = [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
         12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
         7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
         11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
         10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
         11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
         12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
         8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
         11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
         8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
         12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
         8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
         8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
         11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
         9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
         13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3] 

sp = np.fft.rfft(mydata)
freq = np.fft.rfftfreq(len(mydata), d= 1.0)

plt.subplot(211)
plt.plot(mydata)
plt.subplot(212)
plt.plot(freq, sp, 'r')
plt.show()

I understand that extrapolation can be dangerous and unreliable, but for the purposes of this project, I'm just trying to get a working prediction function that I can graph.

Your help is greatly appreciated.

Providing a graph of the data and a code sample might help people help you. This also might be the wrong site. There is a statistics site stats.stackexchange.com that might be a better fit. Either way you need to show people some sample data and some code you are using to fit it. — Paul, Dec 29 '13 at 01:58
@Paul Thank you. I've added some code and wanted to post a graph but I my reputation was too low. — ccheng21, Dec 29 '13 at 03:21
@unutbu Please check the 2nd section of code I added using numpy.fft. I understand I can use the inverse rfft function on my frequency domain to go back to time domain, but I first need to manipulate my frequency data somehow to get preficted values...that's where I am stuck. Thank you! — ccheng21, Dec 29 '13 at 03:25
Your first code example can never fit the data since it assumes a constant amplitude. Using a Fourier decomposition like in your second example can provide a changing amplitude. To make some extrapolation you need to have some idea how you want your function to look like, or use some machine learning techniques, but both these topics are offtopic here, and you might want to try http://stats.stackexchange.com. — Benjamin Bannier, Dec 29 '13 at 10:32

unutbu · Answer 1 · 2013-12-29T13:59:03.113

Here is a way to interpolate by representing your periodic data as a Fourier Series. The coefficients used in the Fourier Series are obtained by taking the discrete FFT.

I don't recommend this -- you can see below that the interpolation is not what one would consider intuitively as very good -- but since I had mentioned it in the comments, I'll follow through and show some code :)

import numpy as np
import matplotlib.pyplot as plt
import scipy.fftpack as fftpack

def fft_inverse(Yhat, x):
    """Based on http://stackoverflow.com/a/4452499/190597 (mtrw)"""
    Yhat = np.asarray(Yhat)
    x = np.asarray(x).reshape(-1, 1)
    N = len(Yhat)
    k = np.arange(N)
    total = Yhat * np.exp(1j * x * k * 2 * np.pi / N)
    return np.real(total.sum(axis=1))/N

mydata = [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
         12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
         7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
         11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
         10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
         11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
         12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
         8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
         11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
         8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
         12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
         8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
         8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
         11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
         9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
         13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3] 

Yhat = fftpack.fft(mydata)

fig, ax = plt.subplots(nrows=2, sharex=True)
xs = np.arange(len(mydata))
ax[0].plot(xs, mydata)

new_xs = np.linspace(xs.min(), xs.max(), len(mydata)*1.5)
new_ys = fft_inverse(Yhat, new_xs)
ax[1].plot(new_xs, new_ys)

plt.xlim(xs.min(), xs.max())
plt.show()

enter image description here

Here is how to use scipy.optimize to find parameters to fit a model function which can then be used to interpolate at arbitrary x-coordinates. The fit using a single sin is still pretty horrible, but I'll post the code just to show how to use scipy.optimize anyway:

import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize as optimize

mydata = np.array(
    [8.3, 8.3, 8.3, 8.3, 7.2, 7.8, 7.8, 8.3, 9.4, 10.6, 10.0, 10.6, 11.1, 12.8,
     12.8, 12.8, 11.7, 10.6, 10.6, 10.0, 10.0, 8.9, 8.9, 8.3, 7.2, 6.7, 6.7, 6.7,
     7.2, 8.3, 7.2, 10.6, 11.1, 11.7, 12.8, 13.3, 15.0, 15.6, 13.3, 15.0, 13.3,
     11.7, 11.1, 10.0, 10.6, 9.4, 8.9, 8.3, 8.9, 6.7, 6.7, 6.0, 6.1, 8.3, 8.3,
     10.6, 11.1, 11.1, 11.7, 12.2, 13.3, 14.4, 16.7, 14.4, 13.3, 12.2, 11.7,
     11.1, 10.0, 8.3, 7.8, 7.2, 8.0, 6.7, 7.2, 7.2, 7.8, 10.0, 12.2, 12.8,
     12.8, 13.9, 15.0, 16.7, 16.7, 16.7, 15.6, 13.9, 12.8, 12.2, 10.6, 9.0,
     8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 8.9, 10.0, 10.6, 11.1, 12.0, 11.7,
     11.1, 13.0, 13.3, 13.0, 11.1, 10.6, 10.6, 10.0, 10.0, 10.0, 9.4, 9.4,
     8.9, 8.3, 9.0, 8.9, 9.4, 9.0, 9.4, 10.6, 11.7, 11.1, 11.7, 12.8, 12.8,
     12.8, 13.0, 11.7, 10.6, 10.0, 10.0, 8.9, 9.4, 7.8, 7.8, 8.3, 7.8, 8.9,
     8.9, 8.9, 9.4, 10.0, 10.0, 10.6, 11.0, 11.1, 11.1, 12.2, 10.6, 10.0, 8.9,
     8.9, 9.0, 8.9, 8.3, 8.9, 8.9, 9.4, 9.4, 9.4, 8.9, 8.9, 8.9, 9.4, 10.0,
     11.1, 11.7, 11.7, 11.7, 11.7, 12.0, 11.7, 11.7, 12.0, 11.7, 11.0, 10.6,
     9.4, 10.0, 8.3, 8.0, 7.2, 5.6, 6.1, 5.6, 6.1, 6.7, 8.0, 10.0, 10.6, 11.1,
     13.3, 12.8, 12.8, 12.2, 11.1, 10.0, 10.0, 10.0, 10.0, 9.4, 8.3]) 


def fit(x, a, b, c, d):
    return a*np.sin(b*x + c) + d

xs = np.linspace(0, 2*np.pi, len(mydata))

guess = (mydata.ptp()/2, 10, 0, mydata.mean())
fitting_parameters, covariance = optimize.curve_fit(fit, xs, mydata, p0=guess)
a, b, c, d = fitting_parameters
print(a, b, c, d)

fig, ax = plt.subplots(nrows=2, sharex=True)
ax[0].plot(xs, mydata)

new_xs = np.linspace(xs.min(), xs.max(), len(mydata)*1.5)
new_ys = fit(new_xs, a, b, c, d)
ax[1].plot(new_xs, new_ys)

plt.xlim(xs.min(), xs.max())
plt.show()

enter image description here

You may be able to improve the fit by choosing a better model function (in place of fit). What to choose is a matter of creativity and intuition guided by a priori knowledge of your problem domain. What is better depends not only on the goodness of the fit, but also on how simple or complex you wish your model to be, and/or on how much predictive power it has when applied to new data sets.

Polynomials via Taylor expansion could maybe serve as fitting functions over specified domains. — , Jun 06 '17 at 10:11

Python-Predicting/Extrapolating future data given a data set

1 Answers1