Why does MATLAB produce an error when calling a python script with "from tensorflow import keras"?

Question

I have the following python script (test_from_import.py)

from tensorflow import keras
#import tensorflow.keras
from tensorflow.keras import backend as K

that I call from MATLAB (R2018a) with the following code:

testDir = '.....' % Directory of 'test_from_import.py'
addpath(testDir)

% Specify Python Executable Directory.
pcPythonExeDir = 'C:\Users\dmattioli\AppData\Local\Programs\Python\Python37\python.exe';
[ver, exec, loaded] = pyversion(pcPythonExeDir);
pyversion % Print to command line.

% Ensure python-matlab integration code is on matlab path.
pyDir = fullfile(matlabroot, 'toolbox', 'matlab', 'external', 'interfaces', 'python');
addpath(pyDir);

% Directory containing all relevant python libraries.
pyLibraryDir = 'C:\Users\dmattioli\AppData\Local\Programs\Python\Python37\Lib\site-packages';

% Add folders to python system path.
insert(py.sys.path, int64(0), testDir);
insert(py.sys.path, int64(0), pyDir);
insert(py.sys.path, int64(0), pyLibraryDir);

%% Call python script.
py_test_mod = py.importlib.import_module('test_from_import')

% % Using system call instead of matlab-python integration functionality.
% [result, status] = python('test_from_import.py') % Does not return error.

This produces an error message (see bottom of post) tracing back to the "from tensorflow import keras" line at the top.

This error does not occur if/when you:

1. Comment out the first line and uncomment the "import tensorflow.keras" line (the error then shifts to the "from tensorflow.keras import backend as K" line).

2. Run the command "python test_from_import.py" in the command line, or

3. Run the "[result, status] = ..." system call line instead of the "py_test_mod = ..." line (https://www.mathworks.com/matlabcentral/answers/153867-running-python-script-in-matlab), or

For various reasons, I'd prefer to resolve this problem rather than using one of those 3 alternatives.

I installed everything using pip, with tensorflow being the first installation. Versions of software (Windows 10) are:

1. Python 3.6.8 (3.7.3 has the same issue).
2. h5py 2.90
3. Keras 2.2.4
4. Tensorflow 1.14.0

>> py_test_mod = py.importlib.import_module('test_from_import')
Error using h5r>init h5py.h5r (line 145)
Python Error: AttributeError: type object 'h5py.h5r.Reference' has no attribute '__reduce_cython__'
Error in h5r>init h5py._conv (line 21)
Error in __init__><module> (line 36)
from ._conv import register_converters as _register_converters
Error in saving><module> (line 38)
  import h5py
Error in network><module> (line 40)
from tensorflow.python.keras.engine import saving
Error in training><module> (line 42)
from tensorflow.python.keras.engine.network import Network
Error in multi_gpu_utils><module> (line 22)
from tensorflow.python.keras.engine.training import Model
Error in __init__><module> (line 38)
from tensorflow.python.keras.utils.multi_gpu_utils import multi_gpu_model
Error in advanced_activations><module> (line 27)
from tensorflow.python.keras.utils import tf_utils
Error in __init__><module> (line 29)
from tensorflow.python.keras.layers.advanced_activations import LeakyReLU
Error in __init__><module> (line 26)
from tensorflow.python.keras import layers
Error in __init__><module> (line 25)
from tensorflow.python.keras import applications
Error in __init__><module> (line 82)
from tensorflow.python import keras
Error in __init__><module> (line 24)
from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
Error in test_from_import><module> (line 1)
from tensorflow import keras
Error in <frozen importlib>_call_with_frames_removed (line 219)
Error in <frozen importlib>exec_module (line 728)
Error in <frozen importlib>_load_unlocked (line 677)
Error in <frozen importlib>_find_and_load_unlocked (line 967)
Error in <frozen importlib>_find_and_load (line 983)
Error in <frozen importlib>_gcd_import (line 1006)
Error in __init__>import_module (line 127)
    return _bootstrap._gcd_import(name[level:], package, level)

Answer 1

According to this issue in the h5py repository, the problem is some version incompatibility. The solution that worked for several people was downgrading to h5py v2.8.0.

Installing a specific version using pip can be done on Windows using:

pip install h5py==2.8.0 --force-reinstall

Some additional information about using pip in this context can be found in this Q&A.