Pandas string.contains doesn't work if searched string contains the substring at the beginning of the string

Question

I'm using str.contains to search for rows where the column contains a particular string as a substring

df[df['col_name'].str.contains('find_this')]

This returns all the rows where 'find_this' is somewhere within the string. However, in the rare but important case where the string in df['col_name'] STARTS with 'find_this', this row is not returned by the above query.

str.contains() returns false where it should return true.

Any help would be greatly appreciated, thanks!

EDIT I've added some example data as requested. Image of dataframe. I want to update the 'Eqvnt_id' column, so for example, the rows where column 'Course_ID' contains AAS 102 all have the same 'Eqvnt_id' value.

To do this I need to be able to search the strings in 'Course_ID' for 'AAS 102' in order to locate the appropriate rows. However, when I do this:

df[df['Course_ID'].str.contains('AAS 102')]

The row that has 'AAS 102 (ENGL 102, JST 102, REL 102)' does not appear in the query!

The datatypes are all objects. I've tried mapping them and applying them to string type, but it has had no effect on the success of the query.

The data from the image can be found at https://github.com/isaachowen/stackoverflowquestionfiles

Can you add some sample data with your issue? Because I think this is some data related problem. — jezrael, Feb 09 '20 at 06:22
This should work. You may be having some data encoding issues. Have a check — Chayan Bansal, Feb 09 '20 at 06:32
@jezrael and Chayan, I added an example. I'm not sure what data encoding issues you have in mind of, any thoughts? — cdrootsudormstardashr, Feb 09 '20 at 17:49
One idea - there should be 2 or more whitespaces or tabulator in real data, the best convert column to list and check — jezrael, Feb 09 '20 at 17:54
@jezrael Unfortunately that's not the case. I've made another column of a comma separated list (still a string) of the values and the same issue exists. The example would just be "AAS 102,ENGL 102,JST 102,REL 102" — cdrootsudormstardashr, Feb 09 '20 at 17:58
@jezrael I updated the dataframe to illustrate the unsolved issue. I'm using '/' instead of ',' but the effect is the same — cdrootsudormstardashr, Feb 09 '20 at 18:01
hmmm, another idea is test `df[df['Course_ID'].str.contains('AAS')]` and then `df[df['Course_ID'].str.contains('102')]`, because some data related problem. — jezrael, Feb 09 '20 at 18:45
Please include the output of `df.to_csv()` for the fragment shown in the screenshot. — DYZ, Feb 09 '20 at 21:56
@DYZ added the data to my github, the link is included above — cdrootsudormstardashr, Feb 10 '20 at 19:52
That’s nuts. I even tried importing the CSV that I posted, but same problem. Could it be the version of python I’m using? I don’t know how to diagnose this — cdrootsudormstardashr, Feb 10 '20 at 22:33
@DYZ what version of python are you using? I'm using 3.7.3. Could that be the issue? — cdrootsudormstardashr, Feb 11 '20 at 18:27

score 0 · Accepted Answer · answered Feb 12 '20 at 15:56

TLDR: Experiment with pandas.Series.str.normalize(), trying different Unicode forms until the issue is solved. 'NFKC' worked for me.

The problem had to do with the format of the data in the column that I was doing the...

df['column'].str.contains('substring')

...operation on. Using the pandas.Series.str.normalize() function works. Link here. Sometimes, under some circumstances that I can't deliberately recreate, the strings would have '\xa0' and '\n' appended to them at the beginning or the end of the string. This post helps address how to deal with that problem. Following that post, I for-looped through every string column and changed the unicode form until I found something that worked: 'NFKC'.

score -2 · Answer 2 · answered Feb 09 '20 at 06:28

-2

you can use pandas.Series.str.find() instead - it returns the index where the string is found - if its at the start, the returned index would be 0. If a string is not found, it returns -1.

df[df['col_name'].str.find('find_this') != -1]

Let me know if this helps!

answered Feb 09 '20 at 06:28

Ramsha Siddiqui

460
6
20

Any explanation for why this answer has been down-voted? – Ramsha Siddiqui Feb 09 '20 at 06:48
I'm not sure why this is downvoted, though when I tried the code I got the same problem. Some people are saying it may be a data type issue, though I'm not sure how I can change the data in the column to work properly. – cdrootsudormstardashr Feb 09 '20 at 17:51

Pandas string.contains doesn't work if searched string contains the substring at the beginning of the string

2 Answers2