Re-evaluate existing dataframe for NaN

Question

When creating a dataframe using read_csv(), na-like values (like the string 'NA') are coerced to NaN. In the below example, I am instead creating a dataframe using DataFrame() on a dict, and the string 'NA' is preserved. How can I re-evaluate this dataframe so that this (and any other na-like values) are converted to NaN?

from collections import OrderedDict
test = OrderedDict([('totalSize', 82142),
             ('done', True),
             ('records',
              [OrderedDict([('Name', 'ASST SANTI PAOLO E CARLO'),
                            ('BillingStreet', 'NA'),
                            ('BillingCity', 'MILANO'),
                            ('BillingState', 'MI'),
                            ('BillingPostalCode', '20142'),
                            ('BillingCountry', 'ITALY')]),
               OrderedDict([('Name',
                             'A O UNIVERSITARIA OSPEDALI RIUNITI TRIESTE'),
                            ('BillingStreet', 'VIA FARNETO 3'),
                            ('BillingCity', 'TRIESTE'),
                            ('BillingState', None),
                            ('BillingPostalCode', '34142'),
                            ('BillingCountry', 'ITALY')])])])
testdf = pd.DataFrame(test['records'])

Possible duplicate of [How to set a cell to NaN in a pandas dataframe](https://stackoverflow.com/questions/34794067/how-to-set-a-cell-to-nan-in-a-pandas-dataframe) — Severin Pappadeux, Jul 30 '19 at 01:02

score 2 · Accepted Answer · answered Jul 30 '19 at 00:55

2

After covert to pandas there are multiple way

df=df.mask(df.isin(['NA','N/A','na']))
df=df.replace(['NA','N/A','na'],np.nan)

answered Jul 30 '19 at 00:55

BENY

317,841
20
164
234

score 0 · Answer 2 · answered Jul 30 '19 at 00:55

0

One way would be to cast all the 'NA' strings to None after dataframe creation:

# add as many strings that you would want converted
na_replace = {
    'NA': None # or whatever null value you want
}

pd.DataFrame(test['records']).replace(na_replace)

answered Jul 30 '19 at 00:55

datawrestler

1,527
15
17

score 0 · Answer 3 · answered Jul 30 '19 at 00:56

0

Use numpy library and convert it to NAN as below.

import numpy as np
testdf.replace('NA', np.nan)

answered Jul 30 '19 at 00:56

SUN

181
5

Re-evaluate existing dataframe for NaN

3 Answers3