Stack Overflow
Stack Overflow

Questions tagged [autodiff]

Automatic Differentiation (AD) is a set of techniques based on the mechanical application of the chain rule to obtain derivatives of a function given as a computer program.

Automatic Differentiation (AD) is a set of techniques based on the mechanical application of the chain rule to obtain derivatives of a function given as a computer program.
AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations such as additions or elementary functions such as exp().
By applying the chain rule of derivative calculus repeatedly to these operations, derivatives of arbitrary order can be computed automatically, and accurate to working precision.

Conceptually, AD is different from symbolic differentiation and approximations by divided differences.

AD is used in the following areas:

  • Numerical Methods
  • Sensitivity Analysis
  • Design Optimization
  • Data Assimilation & Inverse Problems

Home page: http://www.autodiff.org/

94 questions
61
votes
3 answers

Why do we call .detach() before calling .numpy() on a Pytorch Tensor?

It has been firmly established that my_tensor.detach().numpy() is the correct way to get a numpy array from a torch tensor. I'm trying to get a better understanding of why. In the accepted answer to the question just linked, Blupon states that: You…
Josiah Yoder
  • 3,321
  • 4
  • 40
  • 58
28
votes
2 answers

Using derivatives as functions in CppAD

I am trying to modify the example here: # include namespace { // --------------------------------------------------------- // define the template function JacobianCases in empty namespace template bool…
user650261
  • 2,115
  • 5
  • 24
  • 47
10
votes
1 answer

How can I tell if a tf op has a gradient or not?

I am interested in using a SparseTensor in tensorflow, however, I often get LookupError: No gradient defined for operation ... Apparently gradient computation is not defined for many ops for sparse tensors. Are there any easy ways to check if an…
Taro Kiritani
  • 724
  • 6
  • 24
6
votes
2 answers

Is it ok to call `tape.watch(x)` when `x` is already a `tf.Variable` in TensorFlow?

Consider the following function def foo(x): with tf.GradientTape() as tape: tape.watch(x) y = x**2 + x + 4 return tape.gradient(y, x) The call to tape.watch(x) is necessary if the function is called say as foo(tf.constant(3.14)), but…
Jakub Arnold
  • 85,596
  • 89
  • 230
  • 327
5
votes
1 answer

How does tf.gradients manages complex functions?

I am working with complex-valued neural networks. For Complex-valued neural networks Wirtinger calculus is normally used. The definition of the derivate is then (take into acount that functions are non-Holomorphic because of Liouville's…
J Agustin Barrachina
  • 3,501
  • 1
  • 32
  • 52
5
votes
1 answer

Pytorch - why does preallocating memory cause "trying to backward through the graph a second time"

Suppose I have a simple one-hidden-layer network that I'm training in the typical way: for x,y in trainData: optimizer.zero_grad() out = self(x) loss = self.lossfn(out, y) loss.backward() optimizer.step()…
dkv
  • 6,602
  • 10
  • 34
  • 54
5
votes
2 answers

how tensorflow handles complex gradient?

Let z is a complex variable, C(z) is its conjugation. In complex analysis theory, the derivative of C(z) w.r.t z don't exist. But in tesnsorflow, we can calculate dC(z)/dz and the result is just 1. Here is an example: x =…
zhd.zhang
  • 51
  • 2
4
votes
1 answer

How does pytorch compute derivatives for simple functions?

When we talk about the auto-differentiation in the pytorch, we are usually presented a graphical structures of tensors based on their formulas, and pytorch will compute the gradients by tracing down the graphical tree using chain rules. However, I…
Caprikuarius2
  • 137
  • 1
  • 7
4
votes
1 answer

Getting gradient of vectorized function in pytorch

I am brand new to PyTorch and want to do what I assume is a very simple thing but am having a lot of difficulty. I have the function sin(x) * cos(x) + x^2 and I want to get the derivative of that function at any point. If I do this with one point…
LivingRobot
  • 883
  • 2
  • 18
  • 34
3
votes
1 answer

Autodiff implementation for gradient calculation

I have worked through some papers about the autodiff algorithm to implement it for myself (for learning purposes). I compared my algorithm in test cases to the output of tensorflow and their outputs did not match in most cases. Therefor i worked…
Frobeniusnorm
  • 272
  • 1
  • 8
3
votes
2 answers

Vectorize jax.lax.cond with vmap

Hi why can't I vectorize the condition function to apply for a list of boolean? or is there something else going on here? DK = jnp.array([[True],[True],[False],[True]]) f1 = lambda x: 1 f2 = lambda y: 0 cond = lambda dk: jax.lax.cond(dk,f1,f2) vcond…
Kapil
  • 81
  • 5
3
votes
1 answer

Automatic Differentiation with respect to rank-based computations

I'm new to automatic differentiation programming, so this maybe a naive question. Below is a simplified version of what I'm trying to solve. I have two input arrays - a vector A of size N and a matrix B of shape (N, M), as well a parameter vector…
P JMU
  • 69
  • 3
3
votes
1 answer

Plotting output of ForwardDiff in Julia

I would just like to use the ForwardDiff.jl functionality to define a function and plot its gradient (evaluated using ForwardDiff.gradient). It seems not be working because the output of ForwardDiff.gradient is this weird Dual type thing, and it's…
Conor
  • 691
  • 5
  • 14
3
votes
1 answer

Find gradient of a function: Sympy vs. Jax

I have a function Black_Cox() which calls other functions as shown below: import numpy as np from scipy import stats # Parameters D = 100 r = 0.05 γ = 0.1 # Normal CDF N = lambda x: stats.norm.cdf(x) H = lambda V, T, L, σ: np.exp(-r*T) * N(…
Sandu Ursu
  • 1,181
  • 1
  • 18
  • 28
3
votes
0 answers

How can I add custom gradients in the Haskell autodifferentiation library "ad"?

If I want to give a custom or known gradient for a function, how can I do that in the ad library? (I don't want to autodifferentiate through this function.) I am using the grad function in this library. If the library doesn't provide this feature,…
kye
  • 443
  • 3
  • 8
1
2 3 4 5 6 7