I have a tensorflow model. There are some polynomial operations in the thing with high powers, and some divisions, i.e. things that take small errors and turn them into big ones. In my model, for 99% of points, everything works out just fine. However, 1% of the time, a small error turns into a big error, and my gradient for those few points, about midway through the graph, become inf. Further up the graph, various operations on those infs easily turn into nans. By the time the gradient gets up to my variables, it has fallen apart, and clipping is moot.
Anyhow, what I would like to do is to clip my gradient, or to do something else to mitigate the damage before it gets out of hand. Since my gradient gets evaluated point-wise in the graph, I could even set the gradient for those problematic points to be 0 and things would move along without too much trouble. I know the approximate location where I would need to deal with my exploding gradients.
This solution, unfortunately, only seems to apply to gradients once they've made it all the way back to my variables. Is it possible to clip gradients halfway through a graph?
