Same code, very different accuracy on windows/ubuntu (Keras/Tensorflow)

Question

import pandas as pd
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
from keras.optimizers import Adam
from sklearn.preprocessing import MinMaxScaler

def create_dataset(dataset, datasetClass, look_back):
    dataX, dataY = [], []
    for i in range(len(dataset)-look_back-1):
        a = dataset[i:(i+look_back), 0]
        dataX.append(a)
        dataY.append(datasetClass[:,(i+look_back):(i+look_back+1)])

    return np.array(dataX), np.array(dataY)

def one_hot_encode(dataset):
    data = np.zeros((11, len(dataset)),dtype='int')
    for i in range(len(dataset)):
        data[dataset[i]-1,i] = 1
    return data

#Set a seed for repeatable results
np.random.seed(12)


dataframe = pd.read_csv('time-series.csv', usecols=[1], engine='python')
dataset = dataframe.values
dataset = dataset.astype('float32')

dataframeClass = pd.read_csv('time-series-as-class.csv', usecols=[1], engine='python')
datasetClass = dataframeClass.values
datasetClass = datasetClass.astype('int')

datasetClass = one_hot_encode(datasetClass)

#normalize input vals
scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(dataset)


#separate to test/train
train_size = int(len(dataset) * 0.67)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size, :], dataset[train_size:len(dataset), :]
trainClass, testClass = datasetClass[:, 0:train_size,], datasetClass[:, train_size:len(dataset)]

#set up sliding windows
look_back = 150
trainX, trainY = create_dataset(train, trainClass, look_back)
testX, testY = create_dataset(test, testClass, look_back)


#reformat for proper passing to nn
trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
trainY = np.squeeze(trainY, 2)
testY = np.squeeze(testY, 2)

# create and fit the LSTM network
model = Sequential()
model.add(LSTM(15, input_shape=(1,look_back)))
model.add(Dense(22,activation='tanh'))
model.add(Dropout(0.2))
model.add(Dense(11,activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['categorical_accuracy'])
print(model.summary())
model.fit(trainX, trainY, epochs=90, batch_size=1, verbose=2)
# make predictions
trainPredict = model.predict(trainX)
testPredict = model.predict(testX)

I've run this on Ubuntu and on Windows. Tested on windows with keras v 2.0.4 and 2.0.8, on ubuntu with 2.0.5(latest version available through conda)

The accuracy on windows is at 17% and the categorical crossentropy is at ~2, it slowly converges but it consistently starts there

the accuracy on ubuntu is 98% and the categorical crossentropy appears to be 0, and it doesn't actually change

the only code difference is path to the csv files, the csv files are exactly the same. What could possibly cause such a drastic difference?

Had the difference been a percent or two, I could write it off as dropout/tf initializing randomly, but as is it's too much to be pure chance

edit: the solution proved to be fixing the categorical csv files, though they were utf-8 apparently there's something else required to get them to play nice with linux when they were created in windows. I'm not sure if I'm allowed to mark my own answer as "accepted"

You'll be able to accept your own answer after 2 days. – charlesreid1 Oct 11 '17 at 02:56 — charlesreid1, Oct 11 '17 at 02:56

score 3 · Accepted Answer · answered Oct 10 '17 at 18:37

3

The issue proved to be in the csv files, which were originally ported from windows. though they were saved in utf-8 format, I still needed to go to libreoffice and save them as linux csv files.

In their initial state, they did not fail to load, but did not properly one-hot encode, leading to all one-hot encodings being 0. Apparently that leads to very high accuracy.

answered Oct 10 '17 at 18:37

Eldar M.

387
2
11

1

When you ask a Windows tool to save in UTF-8, it will typically prefix the data with a UTF-8-encoded byte-order-mark, and it will likely use CR-LF as line separators. These features can confuse Posix-y tools. I suspect LibreOffice strips the BOM and converts the CR-LF line separators into LF line terminators. Perhaps one of these differences caused the problem. – Adrian McCarthy Oct 10 '17 at 20:49

score 1 · Answer 2 · answered Oct 10 '17 at 17:56

1

np.random.seed(12) needs to be set before you import keras

answered Oct 10 '17 at 17:56

Brendan

101
3

Is that necessary to set seed before importing Keras.? I have set seed after the import and have worked fine. – Sreeram TP Oct 10 '17 at 18:11
@SreeramTP yes: https://github.com/fchollet/keras/issues/2743 https://stackoverflow.com/questions/32419510/how-to-get-reproducible-results-in-keras – Brendan Oct 10 '17 at 18:28
Thank you for your comment, though that did not prove to be the issue I did find that importing tensorflow.set_random_seed and setting that has a strong effect on the initial value of the accuracy metric, but this was caused by csv compatibility between windows and ubuntu – Eldar M. Oct 10 '17 at 18:38

Same code, very different accuracy on windows/ubuntu (Keras/Tensorflow)

2 Answers2