Is it possible to intercept a backwards gradient from some arbitrary layer, modify its values and continue the backpropagation back to the start of network, updating the back gradients of all previous layers based off the modified gradient values you provided?
I know that you can directly modify the gradients themselves before applying the update, but as far as I know, that would only modify the gradients of the specified layer without propagating to previous layers' gradients.