I currently log on to site using POST, I then use POST to access a data-table. When I add a form data payload to the POST accessing the data-table of interest (to change the sort by method of the table) the html returned is for the unsorted table.
referrer = "https://edx.standardandpoors.com/mailbox/jsp/MBIList.jsp"
with requests.Session() as s:
p = s.post(spURL_post, data=spPayload, proxies=proxyDict, verify = True)
q= s.post(referrer, data=dataSort, proxies = proxyDict, verify = True)
dataPageResponse = s.get(referrer, proxies = proxyDict, verify = True)
html = BeautifulSoup(dataPageResponse.content,"lxml")
p
: I log in,
q
: I access the data table on the following page (hopefully sorted)
I have tried verify=True, using POST with the data payload after the first POST to get to the data table, sending the data payload using json.dumps(data_payload_dict)
, as a string, and as the dictionary shown below and I still cannot get the correct (sorted table) HTML.
The sorting data payload that I am submitting is pulled directly from the URL encoded form data shown in the Network Inspector after manually clicking the sort button:
dataSort = {"MailboxPath": "All", "MessageNamePattern": "**",
"MsgNamePattern": "", "MessageId": "", "StartDate": "2018-06-03",
"StartTime": "11%3A57", "StartAMPM": "AM", "EndDate": "2018-07-03",
"EndTime": "11%3A58", "EndAMPM": "AM", "id": "", "OrderBy": "CreatedDateTime",
"lastSortOrder": "CreatedDateTime","ascending": "Yes",
"itemsPerPage": "100","currentPage": "1",
"persistOrder": "Yes"}
Please checkout the form data
per the inspector:
Form Data per Inspector
The only thing I can think of is that I am not submitting the payload data in a format the website is expecting. Here is the top of the Request Headers portion Request Headers. Perhaps I need to encode the form data payload?
Thanks