Keras TensorFlow Hub: Getting started with simple ELMO network

Question

I'm trying to get a simple ELMO model working from TensorFlow hub but, it's turning out to be a challenge.

When I run the my code, I'm getting the error: "Inputs to eager execution function cannot be Keras symbolic tensors, but found [<tf.Tensor 'input_69:0' shape=(None, 10) dtype=string>]"

I think I'm messing up the sequence_length args or the inputs. Can anyone please help me?

import tensorflow as tf
import tensorflow_hub as hub
import re

from tensorflow import keras
import tensorflow.keras
from tensorflow.keras.layers import Input, Dense,Flatten
import numpy as np
import keras.callbacks
import io
from sklearn.model_selection import train_test_split

i = 0
max_cells = 51 #countLines()
x_data = np.zeros((max_cells, 10, 1), dtype='object')
y_data = np.zeros((max_cells, 3), dtype='float32')
seqs = np.zeros((max_cells), dtype='int32')

with io.open('./data/names-sample.txt', encoding='utf-8') as f:
    content = f.readlines()
    for line in content:        
        line = re.sub("[\n]", " ", line)        
        tokens = line.split()

        for t in range(0, min(10,len(tokens))):           
            tkn = tokens[t]        
            x_data[i,t] = tkn
            
        seqs[i] = len(tokens)
        y_data[i,0] = 1
        
        i = i+1

def build_model(): 
    tokens = Input(shape=[10,], dtype=tf.string)
    seq_lens = Input(shape=[], dtype=tf.int32)
    
    elmo = hub.KerasLayer(
        "https://tfhub.dev/google/elmo/3",
        trainable=False,
        output_key="elmo",
        signature="tokens",
    )
    out = elmo({"tokens": tokens, "sequence_len": seqs})
    
    model = keras.Model(inputs=[tokens, seq_lens], outputs=out)
    model.compile("adam", loss="sparse_categorical_crossentropy")
    model.summary()

    return model

x_train, x_test, y_train, y_test = train_test_split(x_data, y_data, test_size=0.70, shuffle=True)

model = build_model()
model.fit(x_train, y_train,validation_data=(x_test, y_test),epochs=1,batch_size=32)

Full Error:

TypeError: An op outside of the function building code is being passed a "Graph" tensor. It is possible to have Graph tensors leak out of the function building context by including a tf.init_scope in your function building code. For example, the following function will fail: @tf.function def has_init_scope(): my_constant = tf.constant(1.) with tf.init_scope(): added = my_constant * 2 The graph tensor has name: input_69:0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "C:\temp\Simon\TempElmoNames.py", line 66, in model = build_model()

File "C:\temp\Simon\TempElmoNames.py", line 56, in build_model out = elmo({"tokens": tokens, "sequence_len": seqs})

File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in call outputs = self.call(cast_inputs, *args, **kwargs)

File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_hub\keras_layer.py", line 229, in call result = f()

File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1081, in call return self._call_impl(args, kwargs)

File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1121, in _call_impl return self._call_flat(args, self.captured_inputs, cancellation_manager)

File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1224, in _call_flat ctx, args, cancellation_manager=cancellation_manager)

File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 511, in call ctx=ctx)

File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\execute.py", line 75, in quick_execute "tensors, but found {}".format(keras_symbolic_tensors))

_SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors, but found [<tf.Tensor 'input_69:0' shape=(None, 10) dtype=string>]

Here are the versions I'm working with: Keras: 2.3.1 TF: 2.0.0 TH-hub: 0.12.0

UPDATE 1: I upgraded Keras (2.6.0) TF (2.6.0) & TF Hub(0.12.0) and changed the build_model method on how the seqs and seq_lens are passed.

def build_model(): 
    tokens = Input(shape=[10,], dtype=tf.string)
    seq_lens = Input(shape=[], dtype=tf.int32)
    
    elmo = hub.KerasLayer(
        "https://tfhub.dev/google/elmo/3",
        trainable=False,
        output_key="elmo",
        signature="tokens",
    )
    out = elmo({"tokens": tokens, "sequence_len": seq_lens})
    
    model = keras.Model(inputs=[tokens, seqs], outputs=out)
    model.compile("adam", loss="sparse_categorical_crossentropy")
    model.summary()

    return model

Now I'm getting the error:

ValueError: Input tensors to a Functional must come from tf.keras.Input. Received: [3 3 2 2 3 3 3 5 3 3 3 2 7 2 2 2 3 2 2 3 3 3 3 3 3 2 3 2 3 2 3 3 2 3 3 2 3 2 2 2 2 3 2 2 3 3 5 3 3 3 0] (missing previous layer metadata).

Answer 1

I don't believe it is a bug rather TF gives us freedom in choosing each method. While we can mix match the layer subclass with keras functional api, I guess we can't make the model subclass work with the Model api of keras. This is where, in my opinion the distinction between eager execution and keras graph mode comes into conflict giving rise to this "SymbolicException".

Making TF aware beforehand what mode it should execute solves it.

Answer 2

Ok finally got it working. The first I did it upgraded:

Keras: 2.2.4
TF: 1.15.0
TF: 0.12.0

Next changed my code to use the right version of ELMO model:

import tensorflow_hub as hub
import tensorflow as tf

elmo = hub.Module("https://tfhub.dev/google/elmo/3", trainable=False)

from tensorflow.keras.layers import Input, Lambda, Bidirectional, Dense, Dropout, Flatten, LSTM
from tensorflow.keras.models import Model

def ELMoEmbedding(input_text):
    return elmo(tf.reshape(tf.cast(input_text, tf.string), [-1]), signature="default", as_dict=True)["elmo"]

def build_model():
    input_layer = Input(shape=(1,), dtype="string", name="Input_layer")    
    embedding_layer = Lambda(ELMoEmbedding, output_shape=(1024, ), name="Elmo_Embedding")(input_layer)
    BiLSTM = Bidirectional(LSTM(128, return_sequences= False, recurrent_dropout=0.2, dropout=0.2), name="BiLSTM")(embedding_layer)
    Dense_layer_1 = Dense(64, activation='relu')(BiLSTM)
    Dropout_layer_1 = Dropout(0.5)(Dense_layer_1)
    Dense_layer_2 = Dense(32, activation='relu')(Dropout_layer_1)
    Dropout_layer_2 = Dropout(0.5)(Dense_layer_2)
    output_layer = Dense(3, activation='sigmoid')(Dropout_layer_2)
    model = Model(inputs=[input_layer], outputs=output_layer, name="BiLSTM with ELMo Embeddings")
    model.summary()
    model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
    return model
elmo_BiDirectional_model = build_model()

import numpy as np
import io
import re
from tensorflow import keras 

i = 0
max_cells = 300
x_data = np.zeros((max_cells, 1), dtype='object')
y_data = np.zeros((max_cells, 3), dtype='float32')

with tf.Session() as session:
    session.run(tf.global_variables_initializer()) 
    session.run(tf.tables_initializer())
    model_elmo = elmo_BiDirectional_model.fit(x_data, y_data, epochs=100, batch_size=5)
    train_prediction = elmo_BiDirectional_model.predict(x_data)