np.isreal behavior different in pandas.DataFrame and numpy.array

Question

I have a array like below

np.array(["hello","world",{"a":5,"b":6,"c":8},"usa","india",{"d":9,"e":10,"f":11}])

and a pandas DataFrame like below

df = pd.DataFrame({'A': ["hello","world",{"a":5,"b":6,"c":8},"usa","india",{"d":9,"e":10,"f":11}]})

When I apply np.isreal to DataFrame

df.applymap(np.isreal)
Out[811]: 
       A
0  False
1  False
2   True
3  False
4  False
5   True

When I do np.isreal for the numpy array.

np.isreal( np.array(["hello","world",{"a":5,"b":6,"c":8},"usa","india",{"d":9,"e":10,"f":11}]))
Out[813]: array([ True,  True,  True,  True,  True,  True], dtype=bool)

I must using the np.isreal in the wrong use case, But can you help me about why the result is different ?

Answer 1

A partial answer is that isreal is only intended to be used on array-like as the first argument.

You want to use isrealobj on each element to get the bahavior you see here:

In [11]: a = np.array(["hello","world",{"a":5,"b":6,"c":8},"usa","india",{"d":9,"e":10,"f":11}])

In [12]: a
Out[12]:
array(['hello', 'world', {'a': 5, 'b': 6, 'c': 8}, 'usa', 'india',
       {'d': 9, 'e': 10, 'f': 11}], dtype=object)

In [13]: [np.isrealobj(aa) for aa in a]
Out[13]: [True, True, True, True, True, True]

In [14]: np.isreal(a)
Out[14]: array([ True,  True,  True,  True,  True,  True], dtype=bool)

That does leave the question, what does np.isreal do on something that isn't array-like e.g.

In [21]: np.isrealobj("")
Out[21]: True

In [22]: np.isreal("")
Out[22]: False

In [23]: np.isrealobj({})
Out[23]: True

In [24]: np.isreal({})
Out[24]: True

---

It turns out this stems from .imag since the test that isreal does is:

return imag(x) == 0   # note imag == np.imag

and that's it.

In [31]: np.imag(a)
Out[31]: array([0, 0, 0, 0, 0, 0], dtype=object)

In [32]: np.imag("")
Out[32]:
array('',
      dtype='<U1')

In [33]: np.imag({})
Out[33]: array(0, dtype=object)

This looks up the .imag attribute on the array.

In [34]: np.asanyarray("").imag
Out[34]:
array('',
      dtype='<U1')

In [35]: np.asanyarray({}).imag
Out[35]: array(0, dtype=object)

I'm not sure why this isn't set in the string case yet...

Answer 2

I think this a small bug in Numpy to be honest. Here Pandas is just looping over each item in the column and calling np.isreal() on it. E.g.:

>>> np.isreal("a")
False
>>> np.isreal({})
True

I think the paradox here has to do with how np.real() treats inputs of dtype=object. My guess is it's taking the object pointer and treating it like an int, so of course np.isreal(<some object>) returns True. Over an array of mixed types like np.array(["A", {}]), the array is of dtype=object so np.isreal() is treating all the elements (including the strings) the way it would anything with dtype=object.

To be clear, I think the bug is in how np.isreal() treats arbitrary objects in a dtype=object array, but I haven't confirmed this explicitly.

Answer 3

There are a couple things going on here. First is pointed out by the previous answers in that np.isreal acts strangely when passed ojbects. However, I think you are also confused about what applymap is doing. Difference between map, applymap and apply methods in Pandas is always a great reference.

In this case what you think you are doing is actually:

df.apply(np.isreal, axis=1)

Which essentially calls np.isreal(df), whereas df.applymap(np.isreal) is essentially calling np.isreal on each individual element of df. e.g

np.isreal(df.A)

array([ True,  True,  True,  True,  True,  True], dtype=bool)

np.array([np.isreal(x) for x in df.A])

array([False, False,  True, False, False,  True], dtype=bool)