When I run code locally and try to fetch data from URL and then parse it to text everything work properly.
When I run exactly the same code on the remote server and try to fetch data from URL error HTTP Error 403: Forbidden
occur
Answers from questions: HTTP error 403 in Python 3 Web Scraping, urllib2.HTTPError: HTTP Error 403: Forbidden helped me when I tried to run it locally and everything work fine.
Do you know what can be different in fetching data from remote server while code is the same(locally and on the server) and way of running code is the same but result is absolutely different?
URL that I want to fetch:
url=https://bithumb.cafe/notice
Code that I was trying to use to fetch data(once it work, second not)
try:
request = urllib.request.Request(url)
request.add_header('User-Agent', 'cheese')
logger.info("request: {}".format(request))
content = urllib.request.urlopen(request).read()
logger.info('content: {}'.format(content))
decoded = content.decode('utf-8')
logger.info('content_decoded: {}'.format(decoded))
return decoded
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''`
second way of fetching data(also work locally but on the remote server not):
class AppURLopener(urllib.request.FancyURLopener):
version = "Mozilla/5.0"
method:
try:
opener = AppURLopener()
response = opener.open(url)
logger.info("request response: {}. response type: {}. response_dict: {}"
.format(response, type(response), response.__dict__))
html_response = response.read()
logger.info("html_Response".format(html_response))
encoding = response.headers.get_content_charset('utf-8')
decoded_html = html_response.decode(encoding)
logger.info('content_decoded: {}'.format(decoded_html))
return decoded_html
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''