1

Can I define my own activation function and use it in the TensorFlow Train API, i.e. the high level API with pre-defined estimators like DNNClassifier?

For example, I want to use this code but replace the activation function tf.nn.tanh with something my own:

tf.estimator.DNNClassifier(
  feature_columns=feature_columns,
  hidden_units=[5,10,5
  n_classes=3, 
  optimizer=tf.train.ProximalAdagradOptimizer(learning_rate=0.01,
                                              l1_regularization_strength=0.0001),
  activation_fn=tf.nn.tanh)
Maxim
  • 52,561
  • 27
  • 155
  • 209
Tirtha
  • 598
  • 5
  • 9
  • What's wrong with `activation_fn` parameter in constructor? What particular custom function do you want to use? – Maxim Nov 09 '17 at 09:18
  • Nothing wrong with that parameter. But what if I want to use some scaled or modified version of some functions other than what is available through tf.nn. list? There are many types of bounded function in this world :) – Tirtha Nov 10 '17 at 18:54

1 Answers1

1

If your custom function can be expressed in terms of built-in tensorflow ops, then it's fairly straightforward. For example:

DNNClassifier(feature_columns=feature_columns,
              ...,
              activation_fn=lambda x: 2*tf.nn.tanh(x)+3*tf.nn.relu(x)+1)

In general, activation_fn can be a callable that accepts a tensor of arbitrary shape (because it'll be applied after each layer). Tensorflow will be able to backpropagate through this expression without any problem.

However, if you want a completely new custom op, not expressible via existing ones, you'll have to register it and compute its gradient manually. See this question for the details.

Maxim
  • 52,561
  • 27
  • 155
  • 209