Getting a webpage using requests is taking too long while dealing with Cloudflare server, but works quickly in browser and curl

Question

I am trying to read a rss feed using feedparser. But the program gets stuck when using requests, feedparser or urlopen. The page I am trying to get is here (mangastream.com rss feed)

I can view it in browser or get it using curl in terminal.

>>> import feedparser
>>> feed = feedparser.parse('http://mangastream.com/rss')

It won't get past line 2.

Edit - Could this be an issue with cloudflare server? DoS/bot protection of some sort? But it works in curl. Not behind proxy or anything.

Edit - Somehow it's working but still takes a lot of time (more than 2-3 mins). It works lot faster with curl.

Cannot reproduce, can you give some more details about your setup? — Jesse Bakker, Jan 22 '17 at 13:20
@JesseBakker Setup? :/ I am running Ubuntu, Python 3.5 inside virtualenv. I have tested with other url ( httpbin.org ) so python process can access the network. — Mark Evans, Jan 22 '17 at 13:25

score 0 · Accepted Answer · edited May 23 '17 at 12:00

0

This looks like an issue with Cloudflare on my side, Replacing hostname with IP address and passing domain name as host header is works.

>>> import requests
>>> requests.get("http://104.28.18.11/rss", headers={'host':'mangastream.com'})
<Response [200]>

Source

edited May 23 '17 at 12:00

Community

1
1

answered Jan 22 '17 at 17:32

Mark Evans

974
1
11
29

Getting a webpage using requests is taking too long while dealing with Cloudflare server, but works quickly in browser and curl

1 Answers1