Setting a variable as the first value in a column, then testing if the value of that variable is in the column returns false

Question

I have the following lines:

lowerTime = minuteDF.iloc[0]['timestamp']
print(lowerTime)
print(minuteDF.iloc[0]['timestamp'])
print(lowerTime == minuteDF.iloc[0]['timestamp'])
print(lowerTime in minuteDF['timestamp'])

which outputs:

2017-12-12 14:30:00
2017-12-12 14:30:00
True
False

Why is the last line false? How can I get it to equate to true?

I believe it is a problem related to pandas sometimes converting between numpy and python datetime objects, or something along those lines.

Edit:

print(lowerTime in minuteDF['timestamp'].values)

also equates to False.

Possible duplicate of [Python Pandas -- why does the \`in\` operator work with indices and not with the data?](https://stackoverflow.com/questions/24841768/python-pandas-why-does-the-in-operator-work-with-indices-and-not-with-the-d) — ctring, Mar 12 '18 at 21:12
I have tried `print(lowerTime in minuteDF['timestamp'].values)`, which doesn't work either. That means it isn't the same issue, doesn't it? — KOB, Mar 12 '18 at 21:16
You're right. I also tried something like `lowerTime == minuteDF['timestamp'].values[0]` and it returns `True`, whereas the `in` operator returns `False`. This seems to also be a numpy thing. Would appreciate if someone else can shed light on this. — ctring, Mar 12 '18 at 21:22
@ctring, yes - I have just printed out the classes for `lowerTime` and `minuteDF.iloc[0]['timestamp']` and they were numpy.datetime64 and pandas.Timestamp, respectively — KOB, Mar 12 '18 at 21:25
Is it possible to give a small code to replicate the issue? May be give the dataframe that you used. — manoj, Mar 12 '18 at 21:27
@manoj The dataframe is pulled from an API for which an authorisation key is needed. I have found the issue anyway. — KOB, Mar 12 '18 at 21:31

score 0 · Answer 1 · answered Mar 12 '18 at 21:31

The error was due to pandas having to convert between numpy.datetime64 objects and pandas.Timestamp objects. datetimes in pandas generally seem to be quite messy.

This was my solution:

timestamps = [pd.Timestamp(np_datetime).to_pydatetime() for np_datetime in minuteDF['timestamp']]
lowerTime = timestamps[0]

Then, as I am changing lowerTime and continuously need to check if it is still a datetime in minuteDF, I can just check:

lowerTime in timestamps

Setting a variable as the first value in a column, then testing if the value of that variable is in the column returns false

1 Answers1