0

I'm trying to scrape this site: https://kfz.virtuelles-rathaus.de/igv2-man/servlet/Internetgeschaeftsvorfaelle

But I can't figure out how to set up a valid request body. The Status Code is 200, so the request itself works. In the response it says it couldn't process to data, because I used the Browser navigation. It wants me to use the Website buttons instead.

I think it is because the payload isn't configured correctly, but I can't get it to work.

I already tried following:

  • Setting the headers manually
  • Using requests_toolbelt.multipart.encoder
  • Creating the boundary for the multipart/data-format myself

The decodedPayload copied from Chrome Dev Tools

WKZ_UNTERSCH_Z: WT
WKZ_ERKENN_Z: SJ
WKZ_ZIFFERN: 454
WKZ_SUCHMERKMAL: NULL
BTN_WKZSUCHE: suchen
ZEITSTEMPEL: 2022040815031191

The encoded Payload copied from Chrome Dev Tools

------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_UNTERSCH_Z"

WT
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_ERKENN_Z"

SJ
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_ZIFFERN"

454
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_SUCHMERKMAL"

NULL
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="BTN_WKZSUCHE"

suchen
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="ZEITSTEMPEL"

2022040815031191
------WebKitFormBoundaryj9dFOsSgrDr5dSwA--

My Code:

import requests
from datetime import datetime


def main():

    payload = {"WKZ_UNTERSCH_Z": "WT",
               "WKZ_ERKENN_Z": "GH",
               "WKZ_ZIFFERN": "454",
               "WKZ_SUCHMERKMAL": "NULL",
               "BTN_WKZSUCHE": "suchen",
               "ZEITSTEMPEL": datetime.strftime(datetime.now(), '%Y%m%d%H%S%M%f')[:-4]}

    url = 'https://kfz.virtuelles-rathaus.de/igv2-man/servlet/Internetgeschaeftsvorfaelle'

    # Initialize Session and get Cookie with session ID
    s = requests.Session()
    r = s.get(f'{url}?MANDANT=08337000&AUFRUF=WKZ')

    r = s.post(
        f'{url}', data=payload, verify=False)

    # Save Response for further scraping
    with open('z_1.html', 'w') as f:
        f.write(str(r.text))


if __name__ == '__main__':
    main()

I already thank you guys in advance for your help

EDIT:

The Body that's created when using the MultipartEncoder:

--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_UNTERSCH_Z"

WT
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_ERKENN_Z"

SJ
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_ZIFFERN"

454
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_SUCHMERKMAL"

NULL
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="BTN_WKZSUCHE"

suchen
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="ZEITSTEMPEL"

2022040909485493
--b3dccffd58a47883c42249db16600856--

xWanna
  • 13
  • 3
  • Check this one out https://stackoverflow.com/questions/12385179/how-to-send-a-multipart-form-data-with-requests-in-python – Hans Apr 08 '22 at 15:31
  • @Hans thank you. It helped me to understand it a bit better, but I still couldn't figure it out. I finally undrstood, that that Boundary doens't need the same format. But the server doens't accept my data. I don't get an error. The server tells me not to use the Browser navigation button, I should use the Buttons provided by the Website. I had the same Problem with another site, there I had to update Session Ids in the payload. This time its another problem – xWanna Apr 08 '22 at 16:46

1 Answers1

0

I found the answer to my problem

I wasn't a problem with the data format, but I used the wrong timestamp.

The solution is to use the timestamp of the previous response and not the current time.

xWanna
  • 13
  • 3