The result of fft in tensorflow is different from numpy

Question

I want to use the fft in tensorflow. But I found the result is different when use the FFT function in numpy and tensorflow respectively. Especially when the size of input array is large

import tensorflow as tf
import numpy as np

aa = tf.lin_space(1.0, 10000.0, 10000)
bb = tf.lin_space(1.0, 10000.0, 10000)
dd = tf.concat([[aa],[bb]],axis = 0)
c_input = tf.complex(dd[0,:], dd[1,:])
Spec = tf.fft(c_input)
sess = tf.Session()
uuu = sess.run(Spec)
print(uuu)

aaa = np.linspace(1.0, 10000.0, 10000)
bbb = aaa + 1j*aaa
ccc = np.fft.fft(bbb)
print(ccc)

The result is

[ 11645833.000000+11645826.j         -544529.875000 -6242453.5j
   -913097.437500  -781089.0625j   ...,     78607.218750  -108219.109375j
    103245.156250  -182935.3125j      214871.765625  -790986.0625j  ]
[ 50005000.00000000+50005000.j         -15920493.78559075+15910493.78559076j
  -7962746.10739718 +7952746.10739719j ...,
   5300163.19893340 -5310163.19893345j
   7952746.10739715 -7962746.10739723j
  15910493.78559067-15920493.78559085j]

So, what can I do to get the same result when I use the fft function in tensorflow?? Thank you for answer

I found that the data type of the output of tf.fft is complex64. But output of np.fft.fft is complex128. Is that the key for this question? How can I solve this problem?

score 10 · Accepted Answer · answered Nov 10 '17 at 13:27

You're right, the difference is exactly in dtype in tensorflow and numpy.

Tensorflow tf.fft forces the input tensor to be tf.complex64, most probably due to GPU op compatiblity. Numpy also hardcodes the array type for FFT. The source code is in native C, fftpack_litemodule.c, where the type is NPY_CDOUBLE - 128-bit, i.e. np.complex128. See this issue for details.

So, I'm afraid there's no simple solution to match them. You can try to define the custom tensorflow op, which applies np.fft.fft, but this would require you to evaluate the gradient manually as well. Or avoid applying FFT to large vectors, so that numerical inaccuracy won't be an issue.

score 4 · Answer 2 · answered Dec 05 '19 at 23:35

I did a bit of investigation and while Maxim's answer that the difference comes down to the different dtype is plausible, I don't think it is correct.

It's true that Numpy uses 64-bit operations for its FFT (even if you pass it a 32-bit Numpy array) whereas Tensorflow uses 32-bit operations. However you can do a 32-bit FFT in Scipy. But even the 32-bit Scipy FFT does not match the Tensorflow calculation.

A small test with a sinusoid with some noise:

import matplotlib.pyplot as plt
import numpy as np
import scipy
import tensorflow as tf

X = np.linspace(0, 1, num=512)
data = np.sin(X * 2 * np.pi * 4) + np.random.uniform(-0.3, 0.3, size=512)
data = data.astype(np.float32)

plt.plot(X, data)

Now take some FFTs and compare:

np_fft = np.fft.rfft(data)

sp_fft = scipy.fftpack.rfft(data)
sp_fft = np.r_[sp_fft[:1], sp_fft[1:-1:2] + sp_fft[2:-1:2] * 1j, sp_fft[-1:]]

input_placeholder = tf.placeholder(tf.float32, [512])
tf_fft = tf.signal.rfft(input_placeholder)
with tf.Session() as sess:
    tf_fft_ = sess.run(tf_fft, feed_dict={input_placeholder: data})

plt.plot(np.abs(sp_fft - tf_fft_), label='Scipy-Tensorflow')
plt.plot(np.abs(sp_fft - np_fft), label='Scipy-Numpy')
plt.plot(np.abs(np_fft - tf_fft_), label='Numpy-Tensorflow')
plt.yscale('log')
plt.xlabel('Frequency bin')
plt.ylabel('Difference')
plt.legend();

It's a bit hard to see, but the difference between Numpy and Tensorflow is comparable to the difference between Scipy and Tensorflow, whereas the difference between Numpy and Scipy is much smaller even though Scipy is doing its operations at 32-bits like Tensorflow is. So there seems to be some additional difference in the Tensorflow FFT implementation beyond the bit-depth of the operation. What that difference is is still unclear to me, though.

It is possible that Tensorflow uses a less precise implementation of the FFT. It is likely that they chose to cut corners for the sake of speed, given that precision is highly irrelevant for deep learning. — Cris Luengo, Jul 17 '20 at 13:34

J Agustin Barrachina · Answer 3 · 2020-07-17T13:48:25.920

0

Indeed, the complex64 is not the only reason. This is tested here:

import tensorflow as tf
import numpy as np

aaa = np.linspace(1.0, 10000.0, 10000)
x = aaa + 1j*aaa
x_tensor = tf.convert_to_tensor(x)

tf_fft = tf.signal.fft(x_tensor)
np_fft = np.fft.fft(x)

print(tf_fft.dtype)
print(np.all(tf_fft.numpy() == np_fft))

Here, the output is:

tf.complex128
False

Showing this time, tensorflow's output is complex128 and yet the results is different.

As I have been showed in the comment sections. This difference is so small it might be neglected and, for practical reasons, you are getting the same results. Therefore, the solution is just to cast the input tensor from complex64 to complex128.

edited Jul 17 '20 at 13:48

answered Jul 17 '20 at 13:00

J Agustin Barrachina

3,501
1
32
52

3

Equality comparisons with floating-point values are pointless. Even reversing the order of a summation will cause rounding errors to be different. Unless you quantify the magnitude of the difference and reason around that, you cannot determine if the results are meaningfully different or not. – Cris Luengo Jul 17 '20 at 13:32
1

Well, in that case then we say it is identical. I believe this might be an acceptable error. Example: `5300163.19893343 -5310163.19893342j vs 5300163.19893342 -5310163.19893342j` – J Agustin Barrachina Jul 17 '20 at 13:46

score 0 · Answer 4 · answered Apr 04 '21 at 19:32

0

It must be because of the choice of N to compute FFT.

tf.signal.fft just takes in the signal and uses a default value of N (not sure what).

np.fft.fft takes in the signal along with a parameter N. Different values of N give different FFT values.

answered Apr 04 '21 at 19:32

ayush thakur

438
3
9

The result of fft in tensorflow is different from numpy

4 Answers4

Linked