0

Im have some issue while running example Keras2DML code in this page. While running the code, i've got this error: 

Traceback (most recent call last):
  File "/home/fregy/kerasplayground/sysml/examplenn.py", line 12, in <module>
    sysml_model = Keras2DML(spark, keras_model,input_shape=(3,224,224))
  File "/usr/local/lib/python2.7/dist-packages/systemml/mllearn/estimators.py", line 909, in __init__
    convertKerasToCaffeNetwork(keras_model, self.name + ".proto")
  File "/usr/local/lib/python2.7/dist-packages/systemml/mllearn/keras2caffe.py", line 201, in convertKerasToCaffeNetwork
    jsonLayers = list(chain.from_iterable(imap(lambda layer: _parseKerasLayer(layer), kerasModel.layers)))
  File "/usr/local/lib/python2.7/dist-packages/systemml/mllearn/keras2caffe.py", line 201, in <lambda>
    jsonLayers = list(chain.from_iterable(imap(lambda layer: _parseKerasLayer(layer), kerasModel.layers)))
  File "/usr/local/lib/python2.7/dist-packages/systemml/mllearn/keras2caffe.py", line 137, in _parseKerasLayer
    ret = { 'layer': { 'name':layer.name, 'type':supportedLayers[layerType], 'bottom':_getBottomLayers(layer), 'top':layer.name, paramName:param[paramName] } }
  File "/usr/local/lib/python2.7/dist-packages/systemml/mllearn/keras2caffe.py", line 112, in _getBottomLayers
    return [ bottomLayer.name for bottomLayer in _getInboundLayers(layer) ]
  File "/usr/local/lib/python2.7/dist-packages/systemml/mllearn/keras2caffe.py", line 70, in _getInboundLayers
    for node in layer.inbound_nodes:  # get inbound nodes to current layer
AttributeError: 'Conv2D' object has no attribute 'inbound_nodes'
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/SocketServer.py", line 230, in serve_forever
    r, w, e = _eintr_retry(select.select, [self], [], [],
AttributeError: 'NoneType' object has no attribute 'select'

Im using Tensorflow-GPU 1.5 , and Keras 2.1.3 .

Fregy
  • 111
  • 1
  • 7

1 Answers1

0

Thanks for trying out Keras2DML. The issue arises because the newer Keras versions renamed the attribute inbound_nodes to _inbound_nodes. This issue was fixed in yesterday's commit: https://github.com/apache/systemml/commit/9c3057a34c84d5bf1c698ad0a5c3c34d90412dbb.

Since you are using TensorFlow-GPU, you may want to check if TF grabs onto most of GPU memory when Keras model is compiled using nvidia-smi. If yes, here are two easy workarounds:

a. Hide GPUs from TF:

os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'
os.environ['CUDA_VISIBLE_DEVICES'] = ''
import tensorflow as tf

b. Or minimize the overhead due to TensorFlow:

from keras.backend.tensorflow_backend import set_session    
tf_config = tf.ConfigProto()
tf_config.gpu_options.allow_growth = True
set_session(tf.Session(config=tf_config))
Niketan
  • 156
  • 2
  • 8
  • Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/SocketServer.py", line 230, in serve_forever r, w, e = _eintr_retry(select.select, [self], [], [], AttributeError: 'NoneType' object has no attribute 'select' – Fregy Feb 02 '18 at 10:33
  • The root cause of the error should be somewhere above, likely from one of the threads invoked by SocketServer. This smells like TF or PySpark related issue as SystemML does not use SocketServer. Can you paste the entire stack which might help debug this issue? – Niketan Feb 02 '18 at 16:11
  • @Fregy I encountered a similar error today. As I guessed, the error is related TensorFlow's threadpool, clearing TensorFlow's session should fix the issue: `from keras import backend as K; K.clear_session()` – Niketan May 04 '18 at 22:21