3

Pandas does not seem to find all elements in a list:

df = pd.DataFrame({"rid": ["125264429", "a"], "id": [1, 2]})
1 in df["id"]                # <- expect True, get True
"125264429" in df["rid"]     # <- expect True, get False
df[df["rid"] == "125264429"] # <- yields result

I am sure there is a perfectly reasonable explanation for this behaviour, but I can't seem to find it. It seems that the last two columns contradict each other. Does it have to do something with the fact that the datatype of the "rid" column is object?

user1965813
  • 671
  • 5
  • 16
  • Whenever you do `x in pd.Series()` you actually look for the value in the indexes. So `1 in df['id']` works because there is an index `1` in `df['id']`. You'd get False for `2 in df['id']`, and you'd get True for `0 in df['rid']` as well. – rafaelc Aug 28 '19 at 14:29
  • `"125264429" in df["rid"].values`Returns True – Florian H Aug 28 '19 at 14:29
  • [This](https://stackoverflow.com/questions/19960077/how-to-implement-in-and-not-in-for-pandas-dataframe) is a good read on this subject – Erfan Aug 28 '19 at 14:33

2 Answers2

6

If use in operator it test not values of Series/column, but index values, docs:

print(1 in df["id"])              # <- expect True, get True
print("125264429" in df["rid"])     # <- expect True, get False

is same like:

print(1 in df["id"].index)              # <- expect True, get True
print("125264429" in df["rid"].index)     # <- expect True, get False

So if convert values to numpy array or list it working like expected:

print(1 in df["id"].values)              # <- expect True, get True
print("125264429" in df["rid"].values)     # <- expect True, get True

print(1 in df["id"].tolist())              # <- expect True, get True
print("125264429" in df["rid"].tolist())     # <- expect True, get True
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252
2

I am not sure what in will do here but definitely not what you want (e.g. asking for 2 in df["id"] returns false as well)

The problem is that you use in not with a List or Set. So you have two options: 

df["rid"].isin(["125264429"]).any()

or 

"125264429" in df["rid"].to_list()

(ok probably about a million more but these are the easy ones I can see)

Lutz
  • 612
  • 3
  • 8