6

I just started learning keras. I am trying to train a non-linear regression model in keras but model doesn't seem to learn much.

#datapoints
X = np.arange(0.0, 5.0, 0.1, dtype='float32').reshape(-1,1)
y = 5 * np.power(X,2) + np.power(np.random.randn(50).reshape(-1,1),3)

#model
model = Sequential()
model.add(Dense(50, activation='relu', input_dim=1))
model.add(Dense(30, activation='relu', init='uniform'))
model.add(Dense(output_dim=1, activation='linear'))

#training
sgd = SGD(lr=0.1);
model.compile(loss='mse', optimizer=sgd, metrics=['accuracy'])
model.fit(X, y, nb_epoch=1000)

#predictions
predictions = model.predict(X)

#plot
plt.scatter(X, y,edgecolors='g')
plt.plot(X, predictions,'r')
plt.legend([ 'Predictated Y' ,'Actual Y'])
plt.show()

enter image description here

what am I doing wrong?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
red5pider
  • 351
  • 1
  • 9
  • 24

2 Answers2

13

Your learning rate is way too high.

Also, irrelevant to your issue, but you should not ask for metrics=['accuracy'], as this is a regression setting and accuracy is meaningless.

So, with these changes:

sgd = SGD(lr=0.001);
model.compile(loss='mse', optimizer=sgd)

plt.legend([ 'Predicted Y' ,'Actual Y']) # typo in legend :)

here are some outputs (results will be different among runs, due to the random element of your y):

enter image description here

enter image description here

desertnaut
  • 57,590
  • 26
  • 140
  • 166
  • thanks for being nice and explaining. Can you suggest me some good resources or tutorials to learn Deep learning. – red5pider Feb 23 '18 at 15:23
  • 4
    [Here](https://github.com/ChristosChristofidis/awesome-deep-learning) is an exhaustive list of courses/tutorials currently available. Personally, I think the combination of Andrew Ng's specialization @ Coursera and the fast.ai course by Jeremy Howard ([review](https://hackernoon.com/fast-ai-what-i-learned-from-lessons-1-3-b10f9958e3ff)) is a great approach (scroll down [here](https://towardsdatascience.com/thoughts-after-taking-the-deeplearning-ai-courses-8568f132153) for a very short comparison). [Stanford CNN](http://cs231n.github.io/) is also great (BTW, upvotes are welcome, too ;) – desertnaut Feb 23 '18 at 15:40
  • Why are the predicted Y a series of piecewise straight lines? – ProfRob Aug 28 '22 at 12:45
0

Just adding this since I had this problem recently. My model wasn't fitting to a polynomial and seemed to only be doing linear regression. After increasing the epochs to 50000, the model began fitting the polynomial properly.

julienbonin
  • 1
  • 1
  • 1
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 25 '22 at 14:36