how to check whether the points are within the polygon shapefile？ pandas

Question

I have a "point.csv" like this, the coordinate system is WGS84(4326)

df = pd.DataFrame("point.csv")


        user_ID day latitude    longitude
0   19998   2011/1/1    35.685067   140.211433
2   19998   2011/1/1    35.684606   140.210978
4   19998   2011/1/1    35.690237   140.231615
6   19998   2011/1/1    35.660564   140.314109
9   19998   2011/1/1    35.68513    140.210979
10  19998   2011/1/2    35.685047   140.211501
13  19998   2011/1/2    35.651321   140.119088
15  19998   2011/1/2    35.684992   140.211178
16  19998   2011/1/3    35.685346   140.211135
20  19998   2011/1/3    35.7845 140.317763
23  19998   2011/1/3    35.759994   140.296097
26  19998   2011/1/3    35.685233   140.210879
27  19998   2011/1/4    35.685178   140.211602
29  19998   2011/1/4    35.548604   139.783602
31  19998   2011/1/4    35.551758   139.786091
33  19998   2011/1/4    35.551758   139.786091
35  19998   2011/1/4    35.537746   139.791026
37  19998   2011/1/4    35.548604   139.783602
41  19998   2011/1/4    35.545373   139.713378
42  19998   2011/1/5    35.545428   139.713465

and I have a polygon (shapefile) like this :

"polygon.shp" (the coordinate system is WGS84(4326))

I want to check whether the points are within (or intersect) the polygon, if yes,df['whether'] =1; if no, df['whether'] = 0

What should I do ?

I try the code like this:

polygon_map = gpd.GeoDataFrame.from_file('polygon.shp')
df['geometry'] = gpd.GeoSeries(list(zip(df['longitude'],df['latitude']))).apply(Point)
df2 = gpd.GeoDataFrame(df.copy())  
df2.crs = {'init': 'epsg:4326'}
df2['wether_area'] = np.where(df2.geometry.within(polygon_map), 1, 0)

But the result shows that there is only one point within the polygon. But Actually, all the points should be within the polygon.

I don't know what's wrong.

Thank you very much!!