I have a website that has this input in the admin page:
<input id="data-role-type-container" type="hidden" data-role-type="None">
What I am scraping in this input is the data-role-type attribute. It has a default value of None.
In a table showing all the roles that a user has, are these checkboxes. If I clicked the checkbox on the member role, then the value for data-role-type would change to member. But when I run the set of code, I still see the default value being printed even though I did check the member role checkbox.
Here's some code that may help in this pastebin link: Element Tag Dictionary Parser
Heres the log when the request happens with what gets printed out as a result:
Starting new HTTP connection (1): 127.0.0.1:5000
Starting new HTTP connection (1): 127.0.0.1:5000
send: b'GET /admin/management/accounts/edit_user/Sergio%20ley/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2MjAzNTkwNDAsImV4cCI6MTYyMDc5MTA0MCwianRpIjoiODYwNjVmMWQtZGY2Mi00ZjVlLTgxYTYtYzBkNjAyZmJiM2M2IiwiaWQiOjEsInJscyI6IiIsInJmX2V4cCI6MTYyMjk1MTA0MH0.FHZG64PKASvPAGJW0iaiKC269iGXF6RfscLmM2_1vE0 HTTP/1.1\r\nHost: 127.0.0.1:5000\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
127.0.0.1 - - [06/May/2021 20:56:29] "←[37mGET /admin/management/accounts/edit_user/Sergio%20ley/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2MjAzNTkwNDAsImV4cCI6MTYyMDc5MTA0MCwianRpIjoiODYwNjVmMWQtZGY2Mi00ZjVlLTgxYTYtYzBkNjAyZmJiM2M2IiwiaWQiOjEsInJscyI6IiIsInJmX2V4cCI6MTYyMjk1MTA0MH0.FHZG64PKASvPAGJW0iaiKC269iGXF6RfscLmM2_1vE0 HTTP/1.1←[0m" 200 -
reply: 'HTTP/1.0 200 OK\r\n'
header: Content-Type: text/html; charset=utf-8
header: Content-Length: 45130
header: Set-Cookie: session=0fe4764a-466a-4a20-a388-f38efb1c4f0c; Expires=Wed, 12-May-2021 03:56:29 GMT; Path=/
header: Server: Werkzeug/1.0.1 Python/3.9.0
header: Date: Fri, 07 May 2021 03:56:29 GMT
http://127.0.0.1:5000 "GET /admin/management/accounts/edit_user/Sergio%20ley/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2MjAzNTkwNDAsImV4cCI6MTYyMDc5MTA0MCwianRpIjoiODYwNjVmMWQtZGY2Mi00ZjVlLTgxYTYtYzBkNjAyZmJiM2M2IiwiaWQiOjEsInJscyI6IiIsInJmX2V4cCI6MTYyMjk1MTA0MH0.FHZG64PKASvPAGJW0iaiKC269iGXF6RfscLmM2_1vE0 HTTP/1.1" 200 45130
http://127.0.0.1:5000 "GET /admin/management/accounts/edit_user/Sergio%20ley/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2MjAzNTkwNDAsImV4cCI6MTYyMDc5MTA0MCwianRpIjoiODYwNjVmMWQtZGY2Mi00ZjVlLTgxYTYtYzBkNjAyZmJiM2M2IiwiaWQiOjEsInJscyI6IiIsInJmX2V4cCI6MTYyMjk1MTA0MH0.FHZG64PKASvPAGJW0iaiKC269iGXF6RfscLmM2_1vE0 HTTP/1.1" 200 45130
None
127.0.0.1 - - [06/May/2021 20:56:29] "←[37mPOST /admin/management/accounts/edit_user/Sergio%20ley/ HTTP/1.1←[0m" 200 -
From my understanding, requests.request is supposed to get the way the HTML document looks like as of the moment I am on the page when the value for the data-role-type attribute was changed. Am I wrong or how can this be fixed?
