I have a list of files containing data from sensors together with some data to identify where the sensor output comes from. I have a dataframe df1 that might look like this:
| Product | Date | Sensor 1 data | Sensor 2 data | Sensor 1 cal. | Sensor 2 cal. |
|---|---|---|---|---|---|
| Product 1 | 23-09-2022 | 12 | 25 | 84 | 32 |
| Product 2 | 23-09-2022 | 56 | 28 | 32 | 65 |
| Product 1 | 23-09-2022 | 95 | 65 | 43 | 91 |
I need to combine the data from all sensors in one long dataframe df_all, but I need to keep the identification data, so I know where my data comes from. Hence, the final dataframe should look something like this:
| Product | Date | Sensor data | Sensors cal. |
|---|---|---|---|
| Product 1 | 23-09-2022 | 12 | 84 |
| Product 2 | 23-09-2022 | 56 | 32 |
| Product 1 | 23-09-2022 | 95 | 43 |
| Product 1 | 23-09-2022 | 25 | 32 |
| Product 2 | 23-09-2022 | 28 | 65 |
| Product 1 | 23-09-2022 | 65 | 91 |
My initial approach was to rename the columns in the original dataframe with
df1.rename(columns={"Sensor 1 data": "Sensor data", "Sensor 1 cal.": "Sensor cal."})
df_all = pd.concat([df_all, df1['Product','Date','Sensor data', 'Sensor cal.']])
df1.rename(columns={"Sensor 2": "Sensor data", "Sensor 2 cal.": "Sensor cal."})
df_all = pd.concat([df_all, df1['Product','Date','Sensor data', 'Sensor cal.']])
But using this approach I would need to re-rename the columns back to their original names as I progress, as I would otherwise end up with columns having the same name.
It should be noted, that the column names can be different in different files as well. So "Sensor 1 data" might be called "Sensor value 1" in a different file.
