Is there a way to avoid while loops using pandas in order to speed up my code?

Question

I'm writing a code to merge several dataframe together using pandas .
Here is my first table :

Index   Values  Intensity   
  1       11      98 
  2       12      855 
  3       13      500
  4       24      140

and here is the second one:

Index   Values  Intensity
  1       21     1000
  2       11     2000
  3       24     0.55
  4       25     500

With these two df, I concanate and drop_duplicates the Values columns which give me the following df :

Index   Values  Intensity_df1  Intensity_df2
  1       11       0                0
  2       12       0                0
  3       13       0                0
  4       24       0                0
  5       21       0                0
  6       25       0                0

I would like to recover the intensity of each values in each Dataframes, for this purpose, I'm iterating through each line of each df which is very inefficient. Here is the following code I use:

m = 0
while m < len(num_df):
    n = 0
    while n < len(df3):
        temp_intens_abs = df[m]['Intensity'][df3['Values'][n] == df[m]['Values']]
        if temp_intens_abs.empty:
            merged.at[n,"Intensity_df%s" %df[m]] = 0
        else:
            merged.at[n,"Intensity_df%s" %df[m]] = pandas.to_numeric(temp_intens_abs, errors='coerce')
        n = n + 1
    m = m + 1

The resulting df3 looks like this at the end:

Index   Values  Intensity_df1  Intensity_df2
  1       11      98             2000
  2       12     855                0
  3       13     500                0
  4       24     140             0.55
  5       21       0             1000
  6       25       0              500

My question is : Is there a way to directly recover "present" values in a df by comparing directly two columns using pandas? I've tried several solutions using numpy but without success.. Thanks in advance for your help.

rename the intensity column to Intensity_df1 Intensity_df2. then apply outer join . Refer this [link](https://stackabuse.com/how-to-merge-dataframes-in-pandas) (search for df_outer) — gowridev, Jun 01 '21 at 10:01
Please read this: https://stackoverflow.com/questions/20109391/how-to-make-good-reproducible-pandas-examples — Be Chiller Too, Jun 01 '21 at 13:59

score 1 · Answer 1 · answered Jun 01 '21 at 09:46

1

You can try joining these dataframes: df3 = df1.merge(df2, on="Values")

answered Jun 01 '21 at 09:46

Be Chiller Too

2,502
2
16
42

Thanks for your answer, but I haven't mentioned that I will also merge more than two table, it can be 3 to 10 df to merge... – julien maillard Jun 01 '21 at 11:29
Where do these dataframes come from? How do you create them? – Be Chiller Too Jun 01 '21 at 11:55
Theses, are datas from a analytical machine. – julien maillard Jun 01 '21 at 13:35
How do you create them? – Be Chiller Too Jun 01 '21 at 13:59

Is there a way to avoid while loops using pandas in order to speed up my code?

1 Answers1