neural network data normalization raises ValueError : Input contains NaN, infinity or value too large for float64

Question

My data set has values in exponential form, like this:

2.15E-06    -0.000556462    0.000197385 -0.000919   -0.000578077....

This is the code:

    ####----Data----####
quest=pd.read_csv("inputFile.csv", names=["A1","A2",..."A200","Sim"])
print(quest.head())
    ####----Set up Data and Label----####
X=quest.drop('Sim',axis=1)
y=quest['Sim']
    ####----Train Test Split----####
X_train, X_test, y_train, y_test = train_test_split(X, y)
np.isfinite(X_train).any(), np.isfinite(y_train).any(),np.isfinite(X_test).any()
np.isnan(X_train).any(), np.isnan(y_train).any(), np.isnan(X_test).any()
    ####----Data Pre-Processing----####
scaler=StandardScaler()
 # Fit only to the training data
X_scaled=scaler.fit(X_train)
 # Now apply the transformations to the data:
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
    ####----Training the Model----####
mlp=MLPClassifier(hidden_layer_sizes=(13,13,13), max_iter=500)
mlp.fit(X_train,y_train)
print(mlp)
    ####----Predictions and Evaluation----####
predictions=mlp.predict(X_test)

print(confusion_matrix(y_test,predictions))
print(classification_report(y_test,predictions))

I got this error:

Traceback (most recent call last): File "E:\thesis\sk-ANN.py", line 67, in

X_scaled=scaler.fit(X_train)...
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

the same error occur while fitting and predicting the model "mlp.fit(X_train,y_train" and "predictions=mlp.predict(X_test)"

how can I remove this error?

Answer 1

I think you need to preprocess your dataset. You can replace the Nan value with some meaningful values. some useful answer can be seen here- numpy array: replace nan values with average of columns