You are using the wrong predicate to check if a point is inside or on the boundary of a polygon.
From documentation on contains
(which is inverse of within
):
object.contains(other)
Returns True
if no points of other
lie in the exterior of the object and at least one point of the
interior of other lies in the interior of object.
Instead, as your point lies on the boundary, you should use intersects
:
object.intersects(other)
Returns True
if the boundary or interior of the object intersect in any way with those of the other.
In other words, geometric objects intersect if they have any boundary
or interior point in common.
(emphasis mine).
Small reproducible example:

>>> from shapely.geometry import Point, Polygon
>>> Polygon([(0, 0), (1, 0), (1, 1), (0, 1)]).contains(Point(1, 1))
False
>>> Point(1, 1).within(Polygon([(0, 0), (1, 0), (1, 1), (0, 1)]))
False
>>> Point(1, 1).intersects(Polygon([(0, 0), (1, 0), (1, 1), (0, 1)]))
True
Note, however, that due to precision errors you indeed can have an unexpected result:

>>> Point(2/3, 2).intersects(Polygon([(0, 0), (1, 0), (1, 3)]))
False
>>> Point(2/3, 2).distance(Polygon([(0, 0), (1, 0), (1, 3)]))
>>> 0.0
In cases like this you might consider either checking the distance to the polygon as shown above, or dilating your polygon a bit using buffer
:
>>> Point(2/3, 2).intersects(Polygon([(0, 0), (1, 0), (1, 3)]).buffer(1e-9))
True