Pandas: Check if column exists in df from a list of columns

Question

Goal here is to find the columns that does not exist in df and create them with null values.

I have a list of column names like below:

column_list = ('column_1', 'column_2', 'column_3')

When I try to check if the column exists, it gives out True for only columns that exist and do not get False for those that are missing.

for column in column_list:
    print df.columns.isin(column_list).any()

In PySpark, I can achieve this using the below:

for column in column_list:
        if not column in df.columns:
            df = df.withColumn(column, lit(''))

How can I achieve the same using Pandas?

You never use your `column` variable in your for loop. You are doing the same operation repeatedly.. basically checking if there is `any` column in `df` that Is in `column_list` — rafaelc, Oct 22 '18 at 17:51

score 11 · Accepted Answer · edited Oct 22 '18 at 18:01

11

Here is how I would approach:

import numpy as np

for col in column_list:
    if col not in df.columns:
        df[col] = np.nan

edited Oct 22 '18 at 18:01

rafaelc

answered Oct 22 '18 at 17:53

quest

2

Note that if for some reason you're averse to importing numpy just for this, numpy is available as `pd.np` – BallpointBen Oct 22 '18 at 17:58
1

Thank you. I was trying to avoid using Numpy as I want to deploy this in AWS Lambda and trying to import as less libraries I can. This was very useful to know. – Raj Oct 22 '18 at 18:04

score 9 · Answer 2 · answered Oct 22 '18 at 17:57

9

Using np.isin, assign and unpacking kwargs

s = np.isin(column_list, df.columns)
df = df.assign(**{k:None for k in np.array(column_list)[~s]})

answered Oct 22 '18 at 17:57

rafaelc

2 Answers2