I have a small project to webscrape prices from some stores using requests and beautiful soup.
One of these stores is now checking for javascript enable browser. I have found that it uses Magento 1.
I have tried the requests-html library, but it did not work.
I know that I can use selenium and headeless-chrome. I have tested it and works fine.
But as long as I run the project on the cloud, it would be much easier and less expensive to use requestes.
On stackoverflow there is a post where one solution was suggested: send the request with the cookied that is set by a website when it checks for javascript enable.
https://stackoverflow.com/a/66917621
Has anyone tried this solution on Magento 1?
I suspect that the data that I need to scrape is not generated by javascript on the page, but the check for javascript enable browser does not allow the page to load.
