Value Errors when inversing normalized data with scikit-learn

Question

I have a recurrent neural network and the input has 3 dimensions. I want to normalize the data before the input with the following part of code:

from sklearn.preprocessing import StandardScaler, MinMaxScaler
import pandas as pd

df_forMinMaxScaling = = pd.read_csv('C:/Users/User1/Data.csv', sep=';')

scaler_minmax = MinMaxScaler()

scaled_minmax_df = scaler_minmax.fit_transform(df_forMinMaxScaling)
scaled_minmax_df = pd.DataFrame(scaled_minmax_df)

data = df.values
data_scaled_standard = scaled_standard_df.values
data_scaled_minmax = scaled_minmax_df.values

Then I train the network. Afterward, I want to use the network on the test data that I created before (that is also normalized) by using the followin part of code:

Y_pred = model6.predict(X_test)

last_list=[]

for i in range (0, len(Y_pred)):
  last_list.append((Y_pred[i][0][steps_forward-1]))
    

actual = pd.DataFrame((X_test[:,0]))
actual.rename(columns = {indexWithYLabels:'actual'}, inplace = True)

where X_test[:,0] has the normalized actual values and Y_pred has the normalized prediction values.

Now I have to inverse the normalization I think because I would like to have real predictions and not normalized predictions. For that purpose I tried the following code:

prediction_notNormalized =scaler_minmax.inverse_transform(Y_pred)

But I get the error message

ValueError: Found array with dim 3. Estimator expected <= 2.

I also tried to inverse the normalization of the actual dataframe with 6 dimensions by using the code:

actual_notNormalized =scaler_minmax.inverse_transform(actual)

But here I also get

ValueError: operands could not be broadcast together with shapes (2096,6) (3,) (2096,6) 

So I guess something is wrong in my approach. Can you tell me how I can inverse the transformation of normalized data for a prediction?

EDIT:

Here is the whole code that I use as this was requested:

import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from tensorflow import keras


#%%  Configure parameters

useStandardizedData = False
useNormalizedData = True
epochs = 10
batch_size = 32

steps_backwards = 7 * 24
steps_forward = 24


#%%  "Reading the data"


dataset_generation = pd.read_csv('C:/Users/Eigene Arbeit/Code/Python/Data/Electricity Price Forecasting/Jan19_to_May21/Original Data/Generation.csv', sep=';', header=0, low_memory=False, infer_datetime_format=True, parse_dates={'datetime':[0]}, index_col=['datetime'])
dataset_generation.index = pd.to_datetime(dataset_generation.index, utc=True)

dataset_load= pd.read_csv('C:/Users/Eigene Arbeit/Code/Python/Data/Electricity Price Forecasting/Jan19_to_May21/Original Data/Load.csv', sep=';', header=0, low_memory=False, infer_datetime_format=True, parse_dates={'datetime':[0]}, index_col=['datetime'])
dataset_load.index = pd.to_datetime(dataset_load.index, utc=True)

dataset_price= pd.read_csv('C:/Users/Eigene Arbeit/Code/Python/Data/Electricity Price Forecasting/Jan19_to_May21/Original Data/Price.csv', sep=';', header=0, low_memory=False, infer_datetime_format=True, parse_dates={'datetime':[0]}, index_col=['datetime'])
dataset_price.index = pd.to_datetime(dataset_price.index, utc=True)



dataset_generation_hourly = dataset_generation.resample('1H').mean()
dataset_load_hourly = dataset_load.resample('1H').mean()

#%%  "Create the dataframe"

#Create new dataframe by using the columns of multiple dataframes
df = pd.concat([dataset_generation_hourly['Solar'], dataset_generation_hourly['Wind Offshore'], dataset_generation_hourly['Wind Onshore'], dataset_load_hourly, dataset_price], axis=1)

#Create a new column in the dataframe that is the sum of mutiple other columns of this dataframe
df['RES'] = df.loc[:,['Solar', 'Wind Offshore', 'Wind Onshore']].sum(axis=1)

#Drop several columns from the dataframe
df.drop(['Solar', 'Wind Offshore', 'Wind Onshore'], axis=1, inplace=True)

#Rearrange the position of the columns in the dataframe
df = df.reindex(['Load','RES','Price'], axis=1)


df_forStandardScaling = df
df_forMinMaxScaling = df

#%%  Scale the input data

scaler_standard = StandardScaler()
scaler_minmax = MinMaxScaler()

scaled_standard_df = scaler_standard.fit_transform(df_forStandardScaling)
scaled_standard_df = pd.DataFrame(scaled_standard_df)

scaled_minmax_df = scaler_minmax.fit_transform(df_forMinMaxScaling)
scaled_minmax_df = pd.DataFrame(scaled_minmax_df)



data = df.values
data_scaled_standard = scaled_standard_df.values
data_scaled_minmax = scaled_minmax_df.values

if useStandardizedData==True:
    data = data_scaled_standard

if useNormalizedData==True:
    data = data_scaled_minmax


#%%   Prepare the input data for the RNN


series_reshaped =  np.array([data[i:i + (steps_backwards+steps_forward)].copy() for i in range(len(data) - (steps_backwards+steps_forward))])

split_fraction_trainingData = 0.70
split_fraction_valifatinData = 0.90

timeslot_x_train_end = int(len(series_reshaped)* split_fraction_trainingData)
timeslot_x_valid_end = int(len(series_reshaped)* split_fraction_valifatinData)

X_train = series_reshaped[:timeslot_x_train_end, :steps_backwards] 
X_valid = series_reshaped[timeslot_x_train_end:timeslot_x_valid_end, :steps_backwards] 
X_test = series_reshaped[timeslot_x_valid_end:, :steps_backwards] 

indexWithYLabels = 2
lengthOfTheYData = len(data)-steps_backwards -steps_forward
Y = np.empty((lengthOfTheYData, steps_backwards, steps_forward))  
for step_ahead in range(1, steps_forward + 1):     
   Y[..., step_ahead - 1] =   series_reshaped[..., step_ahead:step_ahead + steps_backwards, indexWithYLabels]
 
Y_train = Y[:timeslot_x_train_end] 
Y_valid = Y[timeslot_x_train_end:timeslot_x_valid_end] 
Y_test = Y[timeslot_x_valid_end:]



#%%  Build the model and trian it

np.random.seed(42)
tf.random.set_seed(42)

model6 = keras.models.Sequential([
    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 3]),
    keras.layers.SimpleRNN(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(steps_forward))
    #keras.layers.Dense(steps_forward)
])

model6.compile(loss="mean_squared_error", optimizer="adam", metrics=['mean_absolute_percentage_error'])
history = model6.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size,
                    validation_data=(X_valid, Y_valid))



#%%    #Predict the test data
Y_pred = model6.predict(X_test)

last_list=[]

for i in range (0, len(Y_pred)):
  last_list.append((Y_pred[i][0][steps_forward-1]))
    

actual = pd.DataFrame((X_test[:,0]))
actual.rename(columns = {indexWithYLabels:'actual'}, inplace = True)
actual['predictions'] = last_list
actual['difference'] = (actual['predictions'] - actual['actual']).abs()
actual['difference_percentage'] = ((actual['difference'])/(actual['actual']))*100


#%%  Reverse the normalization or standardization

#prediction_notNormalized =scaler_minmax.inverse_transform(Y_pred)
#actprediction_notNormalized =scaler_minmax.inverse_transform(Y_pred)
#lastListed_notNormalized =scaler_minmax.inverse_transform(last_list)

Unfortunately I can't share the data, this is why I just made a screenshot of the dataframe df such that you can see its shape: