Cannot perform pickle in python

Question

    def myModel():
        noOfFilters = 60
        sizeOfFilter1 = (5, 5)
        sizeOfFilter2 = (3, 3)
        sizeOfPool = (2, 2)
        noOfNodes = 500
    
        model = Sequential()
        model.add((Conv2D(6, sizeOfFilter1, input_shape=(32, 32, 1), activation='relu')))
        model.add(MaxPooling2D(pool_size=sizeOfPool))
        model.add((Conv2D(16, sizeOfFilter1, activation='relu')))
        model.add(MaxPooling2D(pool_size=sizeOfPool))
        model.add(Flatten())
        model.add(Dense(120,activation='relu'))
        model.add(Dense(84, activation='relu'))
        model.add(Dropout(0.4))
        model.add(Dense(noOfClasses, activation='softmax'))
        model.compile(Adam(learning_rate=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
        return model
    model = myModel()
    
    history = model.fit(X_train,y_train,validation_data=(X_val,y_val),epochs=5,steps_per_epoch=1000)
    
    pickle_out = open("model_trained.p", "wb")
    pickle.dump(model,pickle_out)
    pickle_out.close()

This is the error i'm getting:

    pickle.dump(model,pickle_out)
    TypeError: can't pickle _thread.RLock objects

I want to save my trained model so that i can use it in a different file. When i try to save the model using pickle, this error pops up. What can i do to remove this error? Or is there an alternate way to save my model?

Can you clarify what machine learning framework you are using? — Joac, Nov 26 '20 at 15:17
Try `with open("model_trained.p", "wb") as pickle_out:`, see if it solves. (instead of managing the object by yourself, and with no `close()`) https://stackoverflow.com/questions/19508703/how-to-open-a-file-through-python — Gulzar, Nov 26 '20 at 15:19
@Gulzar. Why would that affect whether the code caused an error or not? — Frank Yellin, Nov 26 '20 at 15:22
@FrankYellin I am assuming the `close` happens before the `pickle.dump` finishes. — Gulzar, Nov 26 '20 at 15:23
`with open(...) as foo` just calls the inner code and then calls `foo.close()`. The purpose of the `with` is to ensure the file is closed, even if there is an error. There is no asynchronous behavior involved. You're thinking of threads, where the `with pool()...` does do a `join()`. — Frank Yellin, Nov 26 '20 at 15:36

score 4 · Answer 1 · answered Nov 26 '20 at 15:29

Tensorflow adds multithreading/processsing which makes it difficult to pickle. However, they provide a custom save method.

From the docs:

def get_model():
    # Create a simple model.
    inputs = keras.Input(shape=(32,))
    outputs = keras.layers.Dense(1)(inputs)
    model = keras.Model(inputs, outputs)
    model.compile(optimizer="adam", loss="mean_squared_error")
    return model


model = get_model()

# Train the model.
test_input = np.random.random((128, 32))
test_target = np.random.random((128, 1))
model.fit(test_input, test_target)

# Calling `save('my_model')` creates a SavedModel folder `my_model`.
model.save("my_model")

# It can be used to reconstruct the model identically.
reconstructed_model = keras.models.load_model("my_model")

https://www.tensorflow.org/guide/keras/save_and_serialize

Cannot perform pickle in python

1 Answers1