0

I tried to get html code from a site name dcinside in Korea, i am using requests but cannot get html code

and this is my code

import requests
url = "http://gall.dcinside.com/board/lists/?id=bitcoins&page=1"
req = requests.get(url)
print (req)
print (req.content)

but the result was

enter image description here

Why I cannot get html codes even using requests??

Vega
  • 27,856
  • 27
  • 95
  • 103
Pigleg
  • 9
  • 1
  • 1

2 Answers2

5

Most likely they are detecting that you are trying to crawl data dynamically, and not giving any content as a response. Try pretending to be a browser and passing some User-Agent headers.

headers = {
    'User-Agent': 'My User Agent 1.0', 
    'From': 'youremail@domain.com'
}

response = requests.get(url, headers=headers)

# use authentic mozilla or chrome user-agent strings if this doesn't work
hspandher
  • 15,934
  • 2
  • 32
  • 45
1

Take a look at this:

  1. Python Web Crawlers and "getting" html source code

Like the guy said in the aforementioned post, you should use urllib2 which will allow you to easily obtain web resources.

Omer Hen
  • 172
  • 1
  • 12