Pandas DataFrame drop consecutive duplicates

Question

I want to modify drop_duplicates in a such way: For example, I've got DataFrame with rows:

| A header | Another header |
| -------- | -------------- |
| First    | el1            | 
| Second   | el2            |
| Second   | el8            |
| First    | el3            |
| Second   | el4            |
| Second   | el5            |
| First    | el6            |
| Second   | el9            |

And I need not to drop all duplicates, but only consecutive ones. So as a result a want:

| A header | Another header |
| -------- | -------------- |
| First    | el1            | 
| Second   | el2            |
| First    | el3            |
| Second   | el4            |
| First    | el6            |
| Second   | el9            |

Tried to do it with for, but maybe there are better ways

Does this answer your question? [Pandas: Drop consecutive duplicates](https://stackoverflow.com/questions/19463985/pandas-drop-consecutive-duplicates) — Emi OB, Dec 28 '22 at 13:07

score 3 · Accepted Answer · answered Dec 28 '22 at 13:34

You can simply do it by using shift() as follows:

import pandas as pd

df = pd.DataFrame({
    'A header': ['First', 'Second', 'Second', 'First', 'Second', 'Second', 'First', 'Second'],
    'Another header': ['el1', 'el2', 'el8', 'el3', 'el4', 'el5', 'el6', 'el9'],
})

print(df)
"""
  A header Another header
0    First            el1
1   Second            el2
2   Second            el8
3    First            el3
4   Second            el4
5   Second            el5
6    First            el6
7   Second            el9
"""

df2 = df[df['A header'] != df['A header'].shift(1)]

print(df2)
"""
  A header Another header
0    First            el1
1   Second            el2
3    First            el3
4   Second            el4
6    First            el6
7   Second            el9
"""

Using shift(1), you can compare each row with the row's previous row.

For more information, see https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.shift.html

What should I do if I want to check by multiple columns? Because if I am doing df[df[['A header', 'C header']] != df[['A header', 'C header']].shift(1)] , I get Nones — hurricane133, Dec 29 '22 at 09:41
@hurricane133 Then, you can do like this: `df2 = df[(df['A header'] != df['A header'].shift(1)) & (df['C header'] != df['C header'].shift(1))]` — Park, Dec 29 '22 at 10:02

phœnix · Answer 2 · 2022-12-28T13:26:34.797

1

extract dup:

l=[]
for i in range(len(df1)-1):
    if df1['A header'][i]==df1['A header'][i+1] :
        l.append(i+1)

drop dup:

 df1.drop(l, inplace=True)

edited Dec 28 '22 at 13:26

answered Dec 28 '22 at 13:18

phœnix

367
1
9

replace df1 by your dataframe name – phœnix Dec 28 '22 at 13:21

Pandas DataFrame drop consecutive duplicates

2 Answers2