How does PyTorch cast for comparsion?

Question

I encountered the following strange issue and I've no idea why it happens. I believe it is related to how PyTorch casts large values near the power of two. Can anyone give me a hint? Thanks for help!

Answer 1

Turns out that for large value, Pytorch does not record the value, but some value near it, and such approximation beyond what native float should have:

import torch
torch.set_printoptions(precision=10, sci_mode=False)

large_power_of_2 = 2 ** 30
a = torch.zeros(3, 5)

for i in range(-10, 10):
    a[0][0] = large_power_of_2 + i
    print('try to print a[0]. Expected: {}, but real is {}'.format(large_power_of_2 + i, a[0][0]))

output:

try to print a[0]. Expected: 1073741814, but real is 1073741824.0
try to print a[0]. Expected: 1073741815, but real is 1073741824.0
try to print a[0]. Expected: 1073741816, but real is 1073741824.0
try to print a[0]. Expected: 1073741817, but real is 1073741824.0
try to print a[0]. Expected: 1073741818, but real is 1073741824.0
try to print a[0]. Expected: 1073741819, but real is 1073741824.0
try to print a[0]. Expected: 1073741820, but real is 1073741824.0
try to print a[0]. Expected: 1073741821, but real is 1073741824.0
try to print a[0]. Expected: 1073741822, but real is 1073741824.0
try to print a[0]. Expected: 1073741823, but real is 1073741824.0
try to print a[0]. Expected: 1073741824, but real is 1073741824.0
try to print a[0]. Expected: 1073741825, but real is 1073741824.0
try to print a[0]. Expected: 1073741826, but real is 1073741824.0
try to print a[0]. Expected: 1073741827, but real is 1073741824.0
try to print a[0]. Expected: 1073741828, but real is 1073741824.0
try to print a[0]. Expected: 1073741829, but real is 1073741824.0
try to print a[0]. Expected: 1073741830, but real is 1073741824.0
try to print a[0]. Expected: 1073741831, but real is 1073741824.0
try to print a[0]. Expected: 1073741832, but real is 1073741824.0
try to print a[0]. Expected: 1073741833, but real is 1073741824.0

Since the recorded data are the same, the comparison returns True.