python for loop - run time optimization on big datasets

Asked Sep 15 '21 at 12:55

Active Sep 15 '21 at 12:55

Viewed 36 times

I have some logic that checks if the id of a specific row appears in another column of the same dataframe for the same row. If the id appears in df['list_of_approved_id'] then I change the value of the field df['flag'] to 1.

df['flag'] = 0  

for ind in df.index:
  if df['id'][ind] in (df['list_of_approved_id'][ind]):
     df['flag'][ind] = 1

The logic was running fine in the past but now it's much slowe. I'm trying to rewrite this logic into something more pythonic that would be running faster. Looking for help for optimising this logic! Any help is appreciated!

asked Sep 15 '21 at 12:55

Hugo Wolf

Does this answer your question? [Pandas Dataframe Check if column value is in column list](https://stackoverflow.com/questions/47513408/pandas-dataframe-check-if-column-value-is-in-column-list) – bb1 Sep 15 '21 at 13:23
Hi Hugo. Does the 'list_of_approved_id' change per row, or is it the same list of approved id's in each row? – MDan Sep 15 '21 at 13:27
@bb1 checking it now – Hugo Wolf Sep 15 '21 at 13:40
@MDan it changes with every row – Hugo Wolf Sep 15 '21 at 13:40

python for loop - run time optimization on big datasets

0 Answers0