0

given a data frame (df), I would like to determine if the current Last Name contains the previous row Last Name value. I attempted using a for loop where I had an if statement of...

if df['Last Name'][index].str.contains(df['Last Name'][index-1]):
    print('yes')

The code above is incorrect. What is the best way of finding if a value of a data frame at a certain index contains another value of a data frame at a different index? I am not looking for if that value is in the column, I am only looking to see if it exists in previous row of the column.

Edit: I am not finding if both values are equal, I want to find if the current row values contains previous row value.

jcjurevis
  • 1
  • 2

2 Answers2

3

You can use pandas' shift to move Last Name by by one row and check if they equal:

df['Last Name shifted'] = df['Last Name'].shift(periods=1)
equals_df = (df['Last Name shifted'] == df['Last Name shifted'])

Now equals_df is a boolean series with True / False - you can do whatever you want with it (check indices with 'True', check how many are Trues etc)

Roim
  • 2,986
  • 2
  • 10
  • 25
  • This could be good approach. Based on similar logic, you can add a new column to frame with LAG function in pyspark as shown here (https://stackoverflow.com/questions/45011320/how-to-get-data-of-previous-row-in-apache-spark) You can then compare this column against your current row. – NNM Sep 26 '22 at 15:45
  • I need to see if the value is contained, not equal. For instance, if the Last Name of one row is Smith and the next row is Doe-Smith. I want to see if Doe-Smith contains Smith. Just comparing 2 rows – jcjurevis Sep 26 '22 at 15:46
  • Then use compare instead of `==`. Still shifting and using whatever operation you want will work – Roim Sep 26 '22 at 16:21
0

Solved: if df['Last Name'][index] in df['Last Name'][index-1]:

jcjurevis
  • 1
  • 2
  • If you want to compare a part of string in the string, you can actually do df['Last Name'].str.contains('Smith') –  Sep 26 '22 at 16:03
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Sep 30 '22 at 09:17