1

Basically I have a custom call method that computes some interesting intermediate values that are important enough to track when training, but not important enough to provide as an additional return value. My model looks something like

class MyModel(tf.keras.Model):
    def __init__(self, *args, **kwargs):
        self.intermediate_values: List[numpy.ndarray] = []
        super().__init__(*args, **kwargs)


    def call(self, inputs, training=False):

        # Do some linear algebra with inputs and different layers etc.

        intermediate_tensor = some_intermediate_result

        if training and some_other_condition:
            self.intermediate_values.append(intermediate_tensor.numpy())

        # Do some more stuff with intermediate_tensor and different layers etc.

        return final_result

This of course throws AttributeError: 'Tensor' object has no attribute 'numpy' since, as I understand it, evaluating to numpy is only applicable to "eager tensors" and here intermediate_tensor is technically a node in the model call graph(?or whatever the correct nomenclature is).

Directly storing a copy of intermediate_tensor also of course doesn't solve the problem since its still not an eager tensor.

A callback won't work since this in right in the middle of the call method.

Maybe a custom metric class that gets set/managed entirely internally in the MyModel class?

I just can't seem to figure out how to actually extract the values for later analysis when the graph is executed.

charlestoncrabb
  • 103
  • 1
  • 8
  • https://stackoverflow.com/questions/63869134/converting-tensorflow-tensor-into-numpy-array this could be one solution if the drawbacks are ok for you. – mhenning Jul 22 '23 at 22:42
  • @mhenning thanks for the link, but yeah just executing the whole thing in eager mode is kind of a non-starter. Again its not necessarily required to be numpy, just some way--_any way_-- of getting those values out mid-call is what I need – charlestoncrabb Jul 22 '23 at 23:04
  • 1
    I've never dealt with this challenge before, it seems very non-trivial. https://github.com/tensorflow/tensorflow/issues/28840#issuecomment-1083326444 was an interesting read about non-eager Tensor behavior. If TensorFlow can't give you the functionality you need, there is also always PyTorch, where this would be really easy to implement, I think. – mhenning Jul 23 '23 at 00:16
  • 1
    You always have the option of breaking away from `model.fit` and writing your own training loop, which gives you a lot more control over the details. – xdurch0 Jul 23 '23 at 06:18
  • @xdurch0 yeah one thing I didn't mention is that the `train_step` method is also overridden, I'm not sure how moving the problem to the train step will help though – charlestoncrabb Jul 23 '23 at 22:20
  • Have you tried to rewrite your code with operations compatible with SymbolicTensors? For example use tf.stack to list the tensors etc. – Georgios Livanos Jul 25 '23 at 20:49
  • If you only want to read out a specific layer output, you can also create a model with `model = tf.keras.Model(inputs=input, outputs=[output, layer_x.output])`. In a custom training loop you can still do the backprop with the outputs only, and save the layer output for later. – mhenning Jul 27 '23 at 18:31

1 Answers1

0

Try to resolve your mention error. AttributeError: 'Tensor' object has no attribute 'numpy',

Create the convert_np_function function and change tensor to a numpy arr and adds into the self.intermediate_values list. tf.py_function is used to build a tensorflow operation that call for custom function during graph execution. The tf.numpy_function handles the conversion between tensorflow tensors and numpy arr inside the custom function.

import tensorflow as tf
import numpy as np

class MyModel(tf.keras.Model):
    def __init__(self, *args, **kwargs):
        self.intermediate_values = []
        super().__init__(*args, **kwargs)

    def call(self, inputs, training=False):

        intermediate_tensor = some_intermediate_result

        if training and some_other_condition:
            py_func = lambda t: tf.py_function(self.convert_np_function, [t], [])
            tf.numpy_function(py_func, [intermediate_tensor], [])

        return final_result

    def convert_np_function(self, intermediate_tensor):
        intermediate_value = intermediate_tensor.numpy()
        self.intermediate_values.append(intermediate_value)
Faisal Shahbaz
  • 431
  • 3
  • 12