How to apply if condtion and apply to dataframe

Question

Need to check the 'detected' key of bool3_res with key is_doc1 of bool_res and bool_2 res

if bool3_res['detected'] == bool1_res['is_doc1'] == True then my resp has to return
if bool3_res['detected'] == bool2_res['is_doc1'] == True then my resp has to return\

3: else return 'Not valid'

Data frame

user_uid,bool1,bool2,bool3,bool1_res,bool2_res,bool3_res
1001,27452.webp,981.webp,d92e.webp,"{'is_doc1': False, 'is_doc2': True}","{'is_doc1': True, 'is_doc2': True}","{'detected': True, 'count': 1}"
1002,27452.webp,981.webp,d92e.webp,"{'is_doc1': True, 'is_doc2': True}","{'is_doc1': False, 'is_doc2': True}","{'detected': True, 'count': 1}"

My code

def new_func(x):
    d1 = df['bool1_res'].to_dict()
    d1 = eval(d1[0])
    d2 = df['bool2_res'].to_dict()
    d2 = eval(d2[0])
    d3 = df['bool3_res'].to_dict()
    d3 = eval(d3[0])

    if d1['is_doc1'] == d3['detected'] == True:
        resp = {
            "task_id": "uid",
            "group_id": "uid",
            "data": {
            "document1": df['bool1'],
            "document2": df['bool3']
            }
            }

    elif d2['is_doc1'] == d3['detected'] == True:
        resp = {
            "task_id": "user_uid",
            "group_id": "uid",
            "data": {
            "document1": df['bool2'],
            "document2": df['bool3']
            }
            }
    elif d3['detected'] == False:
        resp = 'Not valid'
    else:
        resp = 'Not valid'
    return resp
df['new'] = df.apply(new_func, axis = 1)
#df['new'] = df[['bool1', 'bool2', 'bool3', 'bool1_res', 'bool2_res', 'bool3_res']].applymap(new_func)

My expected out

df['new']

{'u_id': 'uid', 'group': 'uid', 'data': {'document1': ['981.webp'], 'document2': {'d92e.webp'}}}"
{'u_id': 'uid', 'group': 'uid', 'data': {'document1': ['27452.webp'], 'document2': {'d92e.webp'}}}"

My Out df['new']

0    {'task_id': 'user_uid', 'group_id': 'uid', 'data': {'document1': ['981.webp', '981.webp'], 'document2': ['d92e.webp', 'd92e.webp']}}
1    {'task_id': 'user_uid', 'group_id': 'uid', 'data': {'document1': ['981.webp', '981.webp'], 'document2': ['d92e.webp', 'd92e.webp']}}
Name: new, dtype: object

Don’t use `eval()`for this. Please share **all** relevant code and data. See: [mcve]. Have you read the Pandas docs? Why are you storing dictionaries in DataFrames? — AMC, Dec 22 '19 at 18:08
@AMC its requied, just execute the code, I have small reproducible code which is working. its an api response, its required to save as data frame — , Dec 23 '19 at 00:18
I'm not sure what you mean by _its requied, just execute the code_. — AMC, Dec 23 '19 at 19:25
@cdac Your expected output shows a key `'u_id'` which is mentioned nowhere in your code. Also, the program output shows `'task_id'` instead. What do you expect `'uid'` to be? Is it supposed to just a fixed string or you want the value of `'user_id'` in that place? — CypherX, Dec 28 '19 at 16:38

score 2 · Answer 1 · answered Dec 27 '19 at 05:54

You should avoid eval and instead use ast.literal_eval with x instead df for processing per rows and for one element lists add [] to x['bool1'], x['bool2'] and x['bool3']:

import ast

def new_func(x):
    d1 = ast.literal_eval(x['bool1_res'])
    d2 = ast.literal_eval(x['bool2_res'])
    d3 = ast.literal_eval(x['bool3_res'])

    if d1['is_doc1'] == d3['detected'] == True:
        resp = {
            "task_id": "uid",
            "group_id": "uid",
            "data": {
            "document1": [x['bool1']],
            "document2": [x['bool3']]
            }
            }
    elif d2['is_doc1'] == d3['detected'] == True:
        resp = {
            "task_id": "user_uid",
            "group_id": "uid",
            "data": {
            "document1": [x['bool2']],
            "document2": [x['bool3']]
            }
            }
    elif d3['detected'] == False:
        resp = 'Not valid'
    else:
        resp = 'Not valid'
    return resp
df['new'] = df.apply(new_func, axis = 1)

print (df['new'].iat[0])
{'task_id': 'user_uid', 'group_id': 'uid', 'data': {'document1': ['981.webp'], 'document2': ['d92e.webp']}}

print (df['new'].iat[1])
{'task_id': 'uid', 'group_id': 'uid', 'data': {'document1': ['27452.webp'], 'document2': ['d92e.webp']}}

Vivian L. · Answer 2 · 2019-12-28T17:27:01.777

I'm assuming this is what your data looks like after expanding your code lines: (Also, it's a lot easier to read if you can even just add some whitespace...^_^)

df = pd.DataFrame(
    [
        [1001, "27452.webp", "981.webp", "d92e.webp",
            "{'is_doc1': False, 'is_doc2': True}",
            "{'is_doc1': True, 'is_doc2': True}",
            "{'detected': True, 'count': 1}"
        ],
        [1002, "27452.webp", "981.webp", "d92e.webp",
            "{'is_doc1': True, 'is_doc2': True}",
            "{'is_doc1': False, 'is_doc2': True}",
            "{'detected': True, 'count': 1}"
        ],
        [1003, "27452.webp", "981.webp", "d92e.webp",
            "{'is_doc1': True, 'is_doc2': True}",
            "{'is_doc1': False, 'is_doc2': True}",
            "{'detected': False, 'count': 1}"
        ],
    ],
    columns=['user_uid', 'bool1', 'bool2', 'bool3', 'bool1_res', 'bool2_res',
             'bool3_res'
    ]
)

My answer

The execution is split into two parts: (1) parsing the strings and (2) processing/making your "new" column values.

# required packages
import ast
import pandas as pd
# for type suggestions
from typing import Any

Part 1: Parse dict strings

This function is applied to each element in your dataframe via pd.DataFrame.applymap and uses ast.literal_eval, as @jezrael rightly suggested.

def str2dict(x: Any):
    """(Step 1) Parses argument using ast.literal_eval"""
    try:
        x = ast.literal_eval(x.strip())

    # if x is not parsable, return x as-is
    except ValueError as e:
        pass

    finally:
        return x

Part 2: Process your data (i.e. make your "new" col)

This function is applied to each row of your dataframe (by pd.DataFrame.agg):

Based on the logic in your posted function, I:

check if bool3['detected'] is False (your first two conditions both have detected == True); if so, raise ValueError
check if is_doc1 is True for bool1, and if not, for bool2
if neither is_doc1 is True, raise ValueError

def make_newcol_entry(x: pd.Series):
    """(Step 2) constructs "new" column value for pandas group"""
    try:
        if x.bool3_res['detected'] is False:
            raise ValueError
        # check is_doc1 properties
        elif x.bool1_res['is_doc1'] is True:
            document1 = x.bool1
        elif x.bool2_res['is_doc1'] is True:
            document1 = x.bool2
        else:
            raise ValueError
    except ValueError:
        entry = "not valid"
        pass
    # if there is `is_doc1` that is True, construct your entry.
    else:
        entry = {
            "task_id": "uid",
            "group_id": "uid",
            "data": {"document1": document1, "document2": x.bool3}
        }

    return entry

To execute, run:

df = df.assign(new=lambda x: x.applymap(str2dict) \
                              .agg(make_newcol_entry, axis=1))

Note that this parses all elements in your dataframe.

To parse only the bool_res columns, you can perform it in two steps:

# select and parse only res cols ('bool#_res'), then apply
df.update(df.filter(regex=r'_res$', axis=1).applymap(str2dict))
df = df.assign(lambda x: x.agg(apply_make_newcol_entry, axis=1))

Result

$ df
    user_uid    bool1   bool2   bool3   bool1_res   bool2_res   bool3_res   new
0   1001    27452.webp  981.webp    d92e.webp   {'is_doc1': False, 'is_doc2': True} {'is_doc1': True, 'is_doc2': True}  {'detected': True, 'count': 1}  {'task_id': 'uid', 'group_id': 'uid', 'data': {'document1': '981.webp', 'document2': 'd92e.webp'}}
1   1002    27452.webp  981.webp    d92e.webp   {'is_doc1': True, 'is_doc2': True}  {'is_doc1': False, 'is_doc2': True} {'detected': True, 'count': 1}  {'task_id': 'uid', 'group_id': 'uid', 'data': {'document1': '27452.webp', 'document2': 'd92e.webp'}}
2   1003    27452.webp  981.webp    d92e.webp   {'is_doc1': True, 'is_doc2': True}  {'is_doc1': False, 'is_doc2': True} {'detected': False, 'count': 1} not valid

$ df['new']
0   {'task_id': 'uid', 'group_id': 'uid', 'data': {'document1': '981.webp', 'document2': 'd92e.webp'}}
1   {'task_id': 'uid', 'group_id': 'uid', 'data': {'document1': '27452.webp', 'document2': 'd92e.webp'}}
2   not valid
Name: new, dtype: object