I have the following setup: Python 3.7 Selenium 3.141.0 Firefox 67.0.4 Geckodriver 0.24.0
I've written a program that scrapes hotel data off of a hotel operator website. Using the link as a query, the program supplies the site with city, check-in and check-out dates, number of persons, etc.
The program starts up firefox and makes the first query and it all goes well. The problem appears from the second query and on because no matter what is provided in the following links it will keep showing the results of the city in the first query, only changing the dates.
When the webdriver is restarted it's back to normal for the first query but beginning with the second one it all goes like before.
I tried using delete_all_cookies() and configuring the webdriver profile to no create any cache but it does not work. I tried using Python to delete all files inside the profile folder and it still does not work.
The strange thing is that if I go in the browser and manually delete "Cookies and other site data" it's ok, but I can't find a way of doing this programmatically. Tried it both in Firefox and Chrome.
Restarting the webdriver also works. I understand that it clears the profile and starts with a fresh one every time. But this is too costly from a time pov.
#First link, it all goes ok
URL = 'https://www.wyndhamhotels.com/en-us/hotels/beijing-china?brand_id=ALL&checkInDate=8/10/2019&checkOutDate=8/11/2019&useWRPoints=false&children=0&adults=2&rooms=1'
DRIVER.get(URL)
# From the second link on, no matter how many searches I d, I always get the results for Beijing
URL = 'https://www.wyndhamhotels.com/en-us/hotels/bremen-germany?brand_id=ALL&checkInDate=9/11/2019&checkOutDate=9/11/2019&useWRPoints=false&children=0&adults=2&rooms=1'
DRIVER.get(URL)
URL = 'https://www.wyndhamhotels.com/en-us/hotels/paris-france?brand_id=ALL&checkInDate=9/11/2020&checkOutDate=9/11/2020&useWRPoints=false&children=0&adults=2&rooms=1'
DRIVER.get(URL)
Is there any way to programmatically delete all cookies and other site data the way that this happens when you do it manually from the menu, while the webdriver is running? Or, in another train of thoughts, what exactly happens when you manually delete cookies and other site data from the browser menu? What gets deleted and from where?
