Negative result for regression using Keras and Tensorflow

Question

I was following the tutorial by Jason Brownlee, and finished section 2. Where his code apparently gave, as a result,

Baseline: 31.64 (26.82) MSE

mine gave

Results: -59.54 (45.45) MSE

I am intrigued by the negative result, since it's supposed to be the price of square meters (the case study is Boston housing data).

His code is as follows:

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# load dataset
dataframe = pandas.read_csv("housing.csv", delim_whitespace=True, header=None)
dataset = dataframe.values
# split into input (X) and output (Y) variables
X = dataset[:, 0:13]
Y = dataset[:, 13]


# define base model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(13, input_dim=13, kernel_initializer='normal',
                    activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model


# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# evaluate model with standardized dataset
estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100,
                           batch_size=5, verbose=0)

kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X, Y, cv=kfold)
print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std()))
print(results)

EDIT: result is now

Using TensorFlow backend. 2017-11-17 15:19:20.550701: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA

Results: -59.57 (45.48) MSE

[ -21.09693904 -34.74104155 -24.24863872 -113.87524421 -80.55262979 > -98.56815988 -39.08693658 -150.61814082 -17.41903878 -15.52491613] Exception ignored in: > Traceback (most recent call last): File "/home/zchihani/pythonEnvs/kerasEnv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 696, in del TypeError: 'NoneType' object is not callable

Interestingly (maybe), further calls did not show the 'exception ignored' part.

--end of edit--

I read here that "when the training error decreases to a certain level, the outputted prediction sometimes is negative".

The proposed response is to use

from keras.constraints import nonneg

but it does not say how or where...

For info, mu keras.json is

{                                                                                
    "backend": "tensorflow",
    "epsilon": 1e-07,
    "floatx": "float64",
    "image_dim_ordering": "tf",
    "image_data_format": "channels_last"
}

and the output of pip list (in the virtual env) gives

autopep8 (1.3.3)
bleach (1.5.0)
enum34 (1.1.6)
flake8 (3.5.0)
h5py (2.7.1)
html5lib (0.9999999)
importmagic (0.1.7)
jedi (0.11.0)
Keras (2.0.9)
Markdown (2.6.9)
mccabe (0.6.1)
numpy (1.13.3)
olefile (0.44)
pandas (0.21.0)
parso (0.1.0)
Pillow (4.3.0)
pip (9.0.1)
pkg-resources (0.0.0)
protobuf (3.4.0)
pycodestyle (2.3.1)
pyflakes (1.6.0)
python-dateutil (2.6.1)
pytz (2017.3)
PyYAML (3.12)
rope (0.10.7)
scikit-learn (0.19.1)
scipy (1.0.0)
setuptools (36.7.0)
six (1.11.0)
tensorflow (1.4.0)
tensorflow-gpu (1.4.0)
tensorflow-tensorboard (0.4.0rc2)
Theano (1.0.0rc1+5.gab69bf0)
Werkzeug (0.12.2)
wheel (0.30.0)
yapf (0.19.0)

Answer 1

Not really an answer, but I need the space.

When I looked into this, I found that the now-depreciated sklearn.cross_validation is apparently supposed to return a negative MSE, based on the idea that greater numbers always indicate better results. See here for a discussion among the developers, especially this comment for a particularly unintuitive result. It's also worth nothing that sklearn.metrics.make_scorer takes a greater_is_better flag that lets the user customize this.

However, the non-depreciated sklearn.model_selection.cross_val_score is not negating MSE, including when I run it. My guess was that the Keras wrapper was messing things up by negating the no-longer-negated MSE, but I can't find any documentation or anything in the Keras code to that effect, and somehow Paddy is running your code and getting the correct result. I tried it again with scikit-learn (0.19.1) and tensorflow (1.4.0), and it's still negative.

So unfortunately my recommendation is to accept that the actual MSE is the negation of what you're getting, or to build your model without Keras and see if that works. You could also try raising an issue with the Keras devs on GitHub.

Answer 2

In fact, this results is not MSE, It's a "score". The loss is what one would like aim to minimise and a score is what one would like to maximise. The difference is here:

https://github.com/keras-team/keras/pull/7788

And here is another similar problem:

scikit-learn cross validation, negative values with mean squared error