0

While migrating to Python from Matlab, I get different results for matrix multiplication and exponentiation.

This is a simple softmax classifier implementation. I run the Python code, export the variables as a mat file, and run the original Matlab code, load the variables exported from Python, and compare them.

Python code:

f = np.array([[4714, 4735, 4697], [4749, 4748, 4709]])
f = f.astype(np.float64)
a = np.array([[0.001, 0.001, 0.001], [0.001, 0.001, 0.001], [0.001, 0.001, 0.001]])

reg = f.dot(a)
omega = np.exp(reg)
sumomega = np.sum(omega, axis=1)

io.savemat('python_variables.mat', {'p_f': f,
                                    'p_a': a,
                                    'p_reg': reg,
                                    'p_omega': omega,
                                    'p_sumomega': sumomega})

Matlab code:

f = [4714, 4735, 4697; 4749, 4748, 4709];
a = [0.001, 0.001, 0.001; 0.001, 0.001, 0.001; 0.001, 0.001, 0.001];

reg = f*a;
omega = exp(reg);
sumomega = sum(omega, 2);
load('python_variables.mat');

I compare the results by checking out the following:

norm(f - p_f) = 0
norm(a - p_a) = 0
norm(reg - p_reg) = 3.0767e-15
norm(omega - p_omega) = 4.0327e-09
norm(omega - exp(p_f*p_a)) = 0

So the difference seems to be caused by the multiplication, and it gets much larger with exp(). And my original data matrix is larger than this. I get much larger values of omega:

norm(reg - p_reg) = 7.0642e-12
norm(omega - p_omega) = 1.2167e+250

This also causes that in some cases sumomega goes to inf or zero in Python but not in Matlab, so the classifier outputs differ.

What am I missing here? How can I fix to get the exact same results?

groove
  • 273
  • 2
  • 7
  • 17
  • 1
    btw, in your real code, are `a` elements exactly the same? – Brenlla Feb 09 '19 at 15:19
  • @Brenlla Yes. Both in Matlab and Python code they are the same. This is the parameter matrix for the softmax classifier, and 0.001s are the initial values. I update them iteratively using numerical optimization. – groove Feb 09 '19 at 16:40
  • 2
    For me, this actually shows, that your result can vary massively just due to floating point precision, which means, that you maybe should rethink your algorithm – user8408080 Feb 09 '19 at 18:25

1 Answers1

3

The difference looks like numerical precision to me. With floating-point operations, the order of operations matter. You get (slightly) different results when reordering operations because the rounding happens differently.

It is likely that Python and MATLAB implement matrix multiplication slightly differently, and therefore you should not expect exactly the same results.

If you need to raise e to the power of the result of this multiplication, you are going to produce a result with a higher degree of imprecision. This is just the nature of floating-point arithmetic.

The issue here is not that you don’t get the exact same result in MATLAB and Python, the issue is that both produce imprecise results, and you are not aware of what precision you are getting.

The softmax function is known to overflow. The solution is to subtract the maximum input value from all input values. See this other question for more details.

Cris Luengo
  • 55,762
  • 10
  • 62
  • 120
  • I was expecting to see some difference because of the inexact floating point representations, but isn't 1e+250 a huge difference? Maybe normalization by dividing omega by sumomega will decrease the difference and both platforms will give the same classification results eventually, but I cannot be sure if they will. – groove Feb 09 '19 at 16:47
  • 2
    @groove: The `exp` function leads to gigantic values. I am not sure why you use it. But these gigantic values, if they differ by a small fraction, will still have a a gigantic absolute difference. Are you sure you need the `exp` there? Where did you get this operation from? I’ve never seen such a thing in classification. – Cris Luengo Feb 09 '19 at 18:18
  • This is the softmax function. `exp(x_i * a_k) / sum_{j=1}^K exp(x_i * a_j)` gives the probability that data point x_i is from the kth class. – groove Feb 09 '19 at 18:50
  • @groove: check out the addition to my answer. – Cris Luengo Feb 09 '19 at 21:13