4

Somewhere along my workflow NaN values in a Pandas DataFrame (filled in using np.Nan) have turned into <NA> values. (I am still trying to figure out how this happened. Reimporting the dataset from a CSV might be responsible?) pandas.DataFrame.dropna works fine. However pandas.DataFrame.isna only maps

NA values, such as None or numpy.NaN [...] Everything else gets mapped to False values.

Is there a way to map NA values of the type pandas._libs.missing.NAType?

fictitious sample

In [1]: import numpy as np
        import pandas as pd

        dictionary = {'environment': ['test', 'prod', 'test', 'prod'], 
                      'event': ['add_rd', 'add_rd', 'add_env', 'add_env'], 
                      'entry': ['yes', np.NaN, 'no', np.NaN]
                     }

        df = pd.DataFrame(dictionary, columns= ['environment', 'event', 'entry'])

(something happes that turns NaN values into <NA> values of the type pandas._libs.missing.NAType)

In [3]: print(df)

          environment    event entry
        0        test   add_rd   yes
        1        prod   add_rd   <NA>
        2        test  add_env    no
        3        prod  add_env   <NA>

Expected output:

In [4]: df["entry"].isna()

Out[4]  0    False
        1     True
        2    False
        3     True
        Name: entry, dtype: bool
marianoju
  • 334
  • 4
  • 15
  • Can you add some data sample and expected ouput? – jezrael Nov 16 '21 at 10:05
  • I can not share the original dataset and I am having trouble recreating the issue. Expected output would be similar to output from `.isna()`. – marianoju Nov 16 '21 at 10:08
  • I ask for sample data. [how to provide a great pandas example](http://stackoverflow.com/questions/20109391/how-to-make-good-reproducible-pandas-examples) should help. – jezrael Nov 16 '21 at 10:09
  • @jezrael I understand what you are asking for. However I fail to produce sample data that behaves like my data (`` values instead of `NaN`). – marianoju Nov 16 '21 at 10:13
  • added sample to question, be free change it lik eneed, add expcted ouput and your code, what you try. – jezrael Nov 16 '21 at 10:15
  • hmmm, so problem is `NA` working for `dropna` but failed for `isna` in real data? – jezrael Nov 16 '21 at 10:38
  • because it should working both or failed both. One reason why failed both is `NA` is string, not missing value – jezrael Nov 16 '21 at 10:39
  • `isna()` fails for `` in real data. As it should according to documentation. I am asking how to map NA values of the type pandas._libs.missing.NAType. Not why `isna()` fails for ``. – marianoju Nov 16 '21 at 10:46
  • hmmm, never seen before `isna` failed for `NA`, can yu show me link to docs? – jezrael Nov 16 '21 at 10:50
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/239271/discussion-between-marianoju-and-jezrael). – marianoju Nov 16 '21 at 10:51
  • 3
    Using `pd.isna(x)` works for me. See [here](https://stackoverflow.com/questions/66495146/natype-object-has-no-attribute-split) – cxxl Feb 02 '22 at 14:27

1 Answers1

1

You can try pd.NA if you're pandas is updated:

df["isna"] = df["entry"].apply(lambda x:True if x is pd.NA else False)
Arya Sadeghi
  • 480
  • 2
  • 16