Python how to get (decoded) html source code

Question

I am trying in python (2.7.13) to get the source code of a webpage (having the current foreign exchange rates). Normally that is no problem with requests.get(url, headers) etc. In this case I can download/get the webpage, but some parts seems to be (base64 ?) encoded.

However when I visit the page in a browser and I view the source code: the right (decoded) code will be shown in the browser. Question is: how can I get the (decoded) web page source. The url is: https://www.isbank.com.tr/en/foreign-exchange-rates

Part of the code I use is:

url = "https://www.isbank.com.tr/en/foreign-exchange-rates"
resp = requests.get(url)
out = resp.text

If possible, can you show some proof of concept of what you tried to do in a code block? — PythonKiddieScripterX, Sep 17 '22 at 08:51
@PyhtonKiddieScripterX thanks. I visited the hyperlink you gave but it is not an utf-8 issue, so 'r.encoding = r.apparent_encoding' didn't help me. Som code added to 1st posting — ni_hao, Sep 17 '22 at 09:09

score 0 · Answer 1 · answered Sep 17 '22 at 09:34

The response contains the text in Turkish, saying that the request is rejected due to the "unusual traffic detected from your device". It seems that the site checks the User-Agent header to prevent simple scripts from crawling it. You can bypass it by adding some plausible header:

url = 'https://www.isbank.com.tr/en/foreign-exchange-rates'
ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
resp = requests.get(url, headers={'User-Agent': ua})
out = resp.text

Python how to get (decoded) html source code

1 Answers1