Deep Neural Network to learn a function f(x) = x^2

Question

Currently, I have a function f(x) = x^2.

I have a dataset, whose feature is x, and the corresponding label is x^2.

I would like my machine learning model to somewhat accurately predict new values.

For example, the prediction of 300 should be close to 300*300 = 90000

In my code, I first create my training data features and labels, which look like features: [0, 1, 2, ... 999] labels: [0, 1, 4, ... 999*999]


import tensorflow as tf
import numpy as np
import logging
import matplotlib.pyplot as plt

logger = tf.get_logger()
logger.setLevel(logging.ERROR)
val = np.empty([1000], dtype = float)
val_squared = np.empty([1000], dtype = float)

#Create training data
for i in range(1000):
    val[i] = i
    val_squared[i] = i*i;

#Create layers of Deep Neural Network
l0 = tf.keras.layers.Dense(units = 500,input_shape=[1])
l1 = tf.keras.layers.Dense(units = 500, activation = 'sigmoid')
l2 = tf.keras.layers.Dense(units = 500, activation = 'sigmoid')
l3 = tf.keras.layers.Dense(units = 1)
model = tf.keras.Sequential([l0, l1, l2, l3])

model.compile(loss='mean_squared_error', optimizer = tf.keras.optimizers.Adam(lr=10))

history = model.fit(val,val_squared,epochs = 500, verbose = False, batch_size = 500)
plt.xlabel('Epoch Number')
plt.ylabel("Loss Magnitude")
plt.plot(history.history['loss'])


print("Prediction of 200: {}".format(model.predict([200.0])))
plt.show()

When the graph is plotted, we can see that the loss converges, which is a sign that the model is learning. However, the actual prediction is very different from our expected value - 332823.16 as opposed to 40000.

The plotted graph can be seen here: https://i.stack.imgur.com/ZB5XQ.jpg

I have tried changing the activation function to relu and tanh, and tweaked hyperparameters to make sure the loss converged, but to no effect. Are there any other ways I can improve the neural network's performance?

Please repeat the intro tour, especially [on topic](https://stackoverflow.com/help/on-topic). Stack Overflow is not a design or review resource. — Prune, Apr 12 '20 at 05:57

score 1 · Accepted Answer · answered Apr 12 '20 at 11:37

Your loss graph is showing an error of around 0.8E11, 8 billion - a very large loss, equivalent to an error of around 300,000 in your predictions.

The reason is probably that your learning rate is 10, which is very high (tf.keras.optimizers.Adam(lr=10)). Normally with Adam one uses a learning rate of around 1e-3 (0.001) or 1e-4 (0.0001).

A couple more points - you shouldn't even need a multi layer model to solve y=x^2, try a single layer model with say 500 hidden nodes to start. Smaller models converge faster.

Deep Neural Network to learn a function f(x) = x^2

1 Answers1