9

I generated some data from a 4th degree polynomial and wanted to create a regression model in Keras to fit this polynomial. The problem is that predictions after fitting seem to be basically linear. Since this is my first time working with neural nets I assume I made a very trivial and stupid mistake.

Here is my code:

model = Sequential()
model.add(Dense(units=200, input_dim=1))
model.add(Activation('relu'))
model.add(Dense(units=45))
model.add(Activation('relu'))
model.add(Dense(units=1))

model.compile(loss='mean_squared_error',
              optimizer='sgd')

model.fit(x_train, y_train, epochs=20, batch_size=50)

loss_and_metrics = model.evaluate(x_test, y_test, batch_size=100)

classes = model.predict(x_test, batch_size=1)

x_train and y_train are numpy arrays containing the first 9900 entries from this file.

I tried different batch_sizes, number of epochs, layer sizes and amounts of training data. Nothing seems to help.

Please point out everything you see that does not make sense!

FloodLuszt
  • 137
  • 2
  • 6

1 Answers1

11

Neural networks generally won't do a good job in extrapolating polynomial functions. However, if your training and testing data are from the same range, you could achieve quite nice results. I generated some data and used your code:

import numpy as np
x_train=np.random.rand(9000)
y_train=x_train**4+x_train**3-x_train
x_train=x_train.reshape(len(x_train),1)

x_test=np.linspace(0,1,100)
y_test=x_test**4+x_test**3-x_test
x_test=x_test.reshape(len(x_test),1)


model = Sequential()
model.add(Dense(units=200, input_dim=1))
model.add(Activation('relu'))
model.add(Dense(units=45))
model.add(Activation('relu'))
model.add(Dense(units=1))

model.compile(loss='mean_squared_error',
              optimizer='sgd')

model.fit(x_train, y_train, epochs=40, batch_size=50, verbose=1)

loss_and_metrics = model.evaluate(x_test, y_test, batch_size=100)

classes = model.predict(x_test, batch_size=1)

test=x_test.reshape(-1)
plt.plot(test,classes,c='r')
plt.plot(test,y_test,c='b')
plt.show()

Note that I increased epochs to 40 to get more iterations and more accurate results. I also set verbose=1 to be able to see how the loss behaves. The loss is indeed decreasing down to 7.4564e-04, and below is the result that I got. The red line is the prediction of the network, and the blue line is the correct value. You can see that they are quite close to each other.

enter image description here

mac13k
  • 2,423
  • 23
  • 34
Miriam Farber
  • 18,986
  • 14
  • 61
  • 76
  • Thank you! For the polynomial that I chose the above code does not really work. But changing the activation form a ReLU to sigmoid(10*x) yields acceptable results. Maybe starting with fitting polynomials for a first shot at neural nets was a poor choice. – FloodLuszt Jul 09 '17 at 20:23
  • 1
    @FloodLuszt in a single layer in 1d Relus are performing piecewise linear regression. You are perhaps better off just using one layer and perhaps increasing the number of nodes significantly. Essentially how well you can fit a curve with line segments depends on the convexity of the function [ so the more the gradient changes, the more nodes you need to add]. see my answer https://stats.stackexchange.com/a/375658/27556 – seanv507 Nov 08 '18 at 18:48