-1

I made a Keras NN model for fake news detection, and I got 89,1 validation accuracy. I used 50 000 samples for training and 10000 for testing and 2000 for validation. I have saved that model. Now I want to load that model, load new data that I want to make prediction based on that data.

import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, RobustScaler, Normalizer, MinMaxScaler
from sklearn.feature_selection import RFE
from sklearn.metrics import accuracy_score

from tensorflow.python.keras.models import Sequential, load_model
from tensorflow.python.keras.layers import Dense, Dropout, LeakyReLU, Conv2D, LSTM, Flatten

from tensorflow.python.keras import optimizers

from tensorflow.python.keras.regularizers import l2
from tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint

import numpy as np



my_model_1 = load_model("keras fake news acc 89.1.h5")

validation_df = pd.read_csv("validation.csv")
validation_features = validation_df.iloc[:,:-1]
validation_results = validation_df.iloc[:,-1].tolist()

scaler = StandardScaler()
validation_features = scaler.transform(validation_features) #ERROR

The problem is that I get an error:

NotFittedError: This StandardScaler instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

If i use fit_transform on my features, I do not get an error, but I get accuracy of 52%, and that's terrible (because I had 89.1 %).

How can I fix this? Do I need to also load the data that I used for training the model, or I can just load a model and pass the data for prediction?

When I trained the model, I used fit_transform for training data and transform for testing data. I guess that now, I should use only transform on my data, but Im getting an error

desertnaut
  • 57,590
  • 26
  • 140
  • 166
taga
  • 3,537
  • 13
  • 53
  • 119
  • 1
    You need to load also the `scaler` used for transforming your features (after you have saved it fist, like your model). – desertnaut Apr 02 '20 at 10:13
  • Loading or importing? I have imported my StandardScaler. I do not know how to load that scaler – taga Apr 02 '20 at 10:14
  • 1
    Loading it, after fitting, as you did with your model: https://stackoverflow.com/questions/41993565/save-minmaxscaler-model-in-sklearn, https://stats.stackexchange.com/questions/392666/regarding-pre-processing-function-standardscaler-in-scikit-learn-library-how-to – desertnaut Apr 02 '20 at 10:14

1 Answers1

1

Save the scaler object while training using either pickle or joblib library. Load this scaler object and then apply transform function on the test data (or real time data).

You trained the model with a data having a different scaling and trying to do predictions on data with different scaling.

Chayan Bansal
  • 1,857
  • 1
  • 13
  • 23
  • Thanks, I did that, but my accuracy is very low, its 52%, and it was 89% when I was making the model – taga Apr 02 '20 at 10:30