Update: issue has been solved and answer should be posted shortly
My goal is to use BeautifulSoup here, if possible, to avoid using Selenium (and timeouts) which has had some issues with refreshing elements when simulating clicking through pagination.
This post has an accepted answer that apparently does what I’m looking for. In that answer, there are shown to be hidden browser request parameters in the html that can be formatted to a URL to load a page as if it were static html.
I attempt to do that on this site, but the elements aren’t being refreshed for each page when trying to access each subsequent pagination link.
In the html, there is a browser request parameter stored in the pagination link elements (the value for “Nao”), which can manually be passed as the second “searchKey” parameter (searchKey=Nao=…) in the browser URL. This works to access each pagination link when done manually (just by incrementing the value Nao value by 10 for each link), but doesn't refresh elements with requests or mechanize modules. The same elements are retrieved each time despite updating the Nao pagination value.
Is it impossible to load elements with BeautifulSoup on a page that is using AJAX? I found an additional ajax=false argument that could be passed from the browser, but this appeared to have no effect.
