1

I have almost the question already posted and answered here:

Perform Download via download button in Python

I also followed the instructions, in the answer of the above link.

In my case I want to download the data from the following page:

https://www.smard.de/home/downloadcenter/download-marktdaten#!?downloadAttributes=%7B%22selectedCategory%22:1,%22selectedSubCategory%22:1,%22selectedRegion%22:%22DE%22,%22from%22:1658872800000,%22to%22:1659563999999,%22selectedFileType%22:%22CSV%22%7D

The download-button is the one called "Datei herunterladen" in the lower right corner.

download button in the lower right corner

I went into the inspect mode downloaded the file and got the following output in the inspection

inspection results

But the resulting

Request URL: 'https://www.smard.de/nip-download-manager/nip/download/market-data'

does not help getting the csv-file. Opened in the browser I get: 'The requested URL was rejected.' On the other hand, it does not even contain the parameters anymore, so it can't be the right download url.

May anyone help to automate this download?

edit Now I also tried

url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
json_body = {'format': "CSV",
'language': "de",
'moduleIds': [1001224, 1004066, 1004067, 1004068, 1001223, 1004069, 1004071, 1004070, 1001226, 1001228, 1001227,1001225],
'region': "DE",
'timestamp_from': 1659304800000,
'timestamp_to': 1659391199999,
'type': "discrete"}

x = requests.post(url, json = json_body)
x.content

> b'Exception when parsing incoming request to JSON object.'

So how do I get the csv-file based on this method?

Thomas R
  • 1,067
  • 11
  • 17

2 Answers2

1

the reason that you're getting "URL Rejected" is because the link specified in the request is not file-specific. The link that the POST request is being made to, contains several files, which are given out based on the body that the request is sent with. If you look into the developer tools, you can see that there is a request payload attached to the request (image attached) request payload. If you want to learn more about sending JSON body data with requests, please take a look at w3 and this thread. I hope that helps!

neiii
  • 140
  • 1
  • 7
  • I checked and tried thi info on w3 and the thread. Apart from that it failed (see the question-edit). I do not understand how to get and save a csv-file based on this. How can I do this? – Thomas R Aug 03 '22 at 11:19
  • @ThomasR you might want to call json.dumps() before you define the body, the end code would look something like this: `url = 'https://www.smard.de/nip-download-manager/nip/download/market-data' json_body = json.dumps({'format': "CSV", 'language': "de", 'moduleIds': [1001224, 1004066, 1004067, 1004068, 1001223, 1004069, 1004071, 1004070, 1001226, 1001228, 1001227,1001225], 'region': "DE", 'timestamp_from': 1659304800000, 'timestamp_to': 1659391199999, 'type': "discrete"}) x = requests.post(url, json = json_body)` – neiii Aug 03 '22 at 11:47
  • with json.dumps() x.content still says: "b'Exception when parsing incoming request to JSON object.'" – Thomas R Aug 03 '22 at 13:05
  • I found my error. (See my answer) Thanks for the hint with the payload. The "json-dumps" was not necessary. – Thomas R Sep 16 '22 at 10:59
0
  1. The json_body of the request-payload was not correct.
  2. And I found an easy way to get the correct one with less effort and less probability to make it wrong.
  • In the inspect mode after the file is downloaded:
  • Click on the "Raw"-Button in the upper right corner of the "request"-tab, than you can get the correct payload per copy and paste.

enter image description here

The final solution is:

url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
payload = {"request_form":[{"format":"CSV","moduleIds":[1001224,1004066,1004067,1004068,1001223,1004069,1004071,1004070,1001226,1001228,1001227,1001225],"region":"DE","timestamp_from":1658872800000,"timestamp_to":1659563999999,"type":"discrete","language":"de"}]}
x = requests.post(url, json = payload)
with open(f'energy_ger.csv', 'wb') as f:
    f.write(x.content)
Thomas R
  • 1,067
  • 11
  • 17