I am trying to get a table from an .aspx page. It is "https://www.mevzuat.gov.tr/p_KulliyatFihrist.aspx". I've tried this and looked at this questions before asking. First one has been solved without requests module and second one is not written in python and also uses selenium library which is not a requests way.
So let me explain what I did: I've imported Session from requests module:
from requests import Session
s = Session()
URL = "https://www.mevzuat.gov.tr/p_KulliyatFihrist.aspx"
firstGET = s.get(URL)
After that I did get the viewstate and other dynamic payload elements like this:
firstGETsoup = BeautifulSoup(firstGET.text,"html.parser")
# parse and retrieve two vital form values
viewstate = firstGETsoup.select("#__VIEWSTATE")[0]['value']
viewstategenerator = firstGETsoup.select("#__VIEWSTATEGENERATOR")[0]['value']
eventvalidation = firstGETsoup.select("#__EVENTVALIDATION")[0]['value']
After getting these values, I've created headers and payload dictionaries:
headers = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0"}
payload = {
"__EVENTTARGET":"ctl00$Body$LinkButton2",
"__EVENTARGUMENT":"",
"__LASTFOCUS":"",
"__VIEWSTATE":viewstate,
"__VIEWSTATEGENERATOR": viewstategenerator,
"__EVENTVALIDATION":eventvalidation,
"ctl00$Body$ddlSeachCriteria":"11", ## This criteria chooses period of time
"ctl00$Body$txtCriteria":"01.01.2000", ## Beginning of period
"ctl00$Body$txtCriteria1":"24.02.2020"} ## End of the period
As you can see four ctl00 elements determined by me for execute search in between of two dates. This search gives 10 posts per page table result. The format of dates is coming directly from network monitor of Mozilla Firefox, so it can't be wrong I suppose.
Finally I've submitted a post request:
post = s.post(URL, data= payload, headers=headers)
After submitting post request, I've submitted get request to retrieve table from the aspx page:
secondGET = s.get(URL)
The id of div that includes table is panelGrid so I've made a "soup" and searched for it:
secondGETsoup = BeautifulSoup(secondGET.text,"html.parser")
g = soup.find_all('table', id='gv')
print(g)
But according to the code above, "g" returns nothing: [ ]
How can I solve this problem? In other words: How can I submit a POST request with custom payloads to .aspx page via python requests module only?
Thanks in advance..
