I have a python code, which loads an already saved Tensorflow model from the disk, re-trains the model and saves it back to disk under different folder. But I'm getting
Segmentation fault error in the save step. I'll paste a part of my code which signals the flow:
# Load an already saved model
model = tf.keras.models.load_model('M1')
# Re-train the model on some newly available dataset.
# ds_train and ds_val are TF datasets
callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=4, restore_best_weights=True)
model.fit(ds_train, epochs=30, validation_data=ds_val, callbacks=[callback])
# Save the re-trained model back to disk
# This is the part of code causing the error
model.save('M2')
The loading of the model, re-training of the model runs smoothly. When it comes to saving the model, I'm getting Segmentation fault. I've pasted the error stacktrace I obtained below after enabling faulthandler
Fatal Python error: Segmentation fault
Current thread 0x00007fa9f06c5740 (most recent call first):
File "lib/python3.5/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 68 in get_resource_handle_data
File "lib/python3.5/site-packages/tensorflow/python/ops/custom_gradient.py", line 65 in copy_handle_data
File "lib/python3.5/site-packages/tensorflow/python/eager/function.py", line 590 in call
File "lib/python3.5/site-packages/tensorflow/python/eager/function.py", line 1938 in _call_flat
File "lib/python3.5/site-packages/tensorflow/python/eager/function.py", line 1848 in _filtered_call
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 846 in _call
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 780 in __call__
File "lib/python3.5/site-packages/tensorflow/python/training/tracking/tracking.py", line 262 in _creator
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 600 in wrapped_fn
File "lib/python3.5/site-packages/tensorflow/python/framework/func_graph.py", line 986 in func_graph_from_py_func
File "lib/python3.5/site-packages/tensorflow/python/eager/function.py", line 3075 in _create_graph_function
File "lib/python3.5/site-packages/tensorflow/python/eager/function.py", line 3213 in _maybe_define_function
File "lib/python3.5/site-packages/tensorflow/python/eager/function.py", line 2855 in _get_concrete_function_internal_garbage_collected
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 697 in _initialize
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 1073 in _get_concrete_function_garbage_collected
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 1167 in get_concrete_function
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 1012 in _list_all_concrete_functions
File "lib/python3.5/site-packages/tensorflow/python/eager/def_function.py", line 1030 in _list_all_concrete_functions_for_serialization
File "lib/python3.5/site-packages/tensorflow/python/saved_model/save.py", line 216 in __init__
File "lib/python3.5/site-packages/tensorflow/python/saved_model/save.py", line 1061 in _build_meta_graph
File "lib/python3.5/site-packages/tensorflow/python/saved_model/save.py", line 976 in save
File "lib/python3.5/site-packages/tensorflow/python/keras/saving/saved_model/save.py", line 80 in save
File "lib/python3.5/site-packages/tensorflow/python/keras/saving/save.py", line 134 in save_model
File "lib/python3.5/site-packages/tensorflow/python/keras/engine/training.py", line 1979 in save
File "model_preparation.py", line 495 in save_model
I'm using python3.8 and tensorflow 2.3.0. I tried with combinations of python3.5 and tensorflow 2.4.0 as well but it is throwing the same error.
Any help would be appreciated.
