why doesn't node-fetch work on this site?

Question

I am trying to scrape the site "https://shmoti.com" . But unfortunately node-fetch's fetch method is not at all getting the response. It works fine for other websites.

Here is my code

const fetch = require("node-fetch")
fetch('https://shmoti.com', options = {headers : {'User-Agent' : 'Mozilla/5.0'}}).then(res=>res.text()).then(res=>console.log(res))

The first promise which resolves the response object is pending all the time.

I have even tried having a User-Agent . I have set a timeout of 60 seconds . I can successfully scrape this site using scrapy library in python but with fetch method , it always times out.

Why is this happening ? How can I fix this ?

I can ping the website and open it in my browser too , but only the fetching from node is not working.

`not at all getting the response` - so, nothing at all? partial response? any errors? do you have error handling in your invisible code to indicate what the problem may be? — Jaromanda X, Jun 29 '18 at 01:54
@Natesh, I tried it here, it works but. https://runkit.com/shankarshastri/node-fetchtest — Shankar Shastri, Jun 29 '18 at 01:59
@ShankarShastri the same code doesn't work when I run it from my machine . Is it possible that the site has blocked me ? If so how can i get around it ? — Natesh bhat, Jun 29 '18 at 02:03
you can change your ip or access via proxy, I can't say much more because I never tried any of those... — Jose Carlos Ramírez Vásquez, Jun 29 '18 at 02:08
https://stackoverflow.com/questions/41111411/node-fetch-only-returning-promise-pending — Shankar Shastri, Jun 29 '18 at 02:18
Then what mechanism might they be using ? I can access their site from chrome normally but the fetch method doesn't work ... :/ My same code works for other websites but not this — Natesh bhat, Jun 29 '18 at 02:18
perhaps you should make the user agent look like an actual real browser — Jaromanda X, Jun 29 '18 at 02:22
Anyway, this exact example works for me, with bad user-agent and everything. (node 8.11.3, node-fetch@2.1.2) — ippi, Jun 29 '18 at 02:24
@ShankarShastri - pinging isn't the same as requesting a page on port 443 :p — Jaromanda X, Jun 29 '18 at 02:26
@ippi - you are right - the page "scrapes" even without a user-agent - perhaps `shmoti` has some anti-scraping logic, or problem exists between keyboard and chair — Jaromanda X, Jun 29 '18 at 02:27
@Nateshbhat - one that prevents people from *borrowing* resources that do not belong to them — Jaromanda X, Jun 29 '18 at 02:28
The scraping works fine when I use scrapy in python without any proxy — Natesh bhat, Jun 29 '18 at 02:29
well, perhaps you're doing something else wrong ... nobody else is having a problem scraping that website using your code - so, the problem is not with the code you've shown — Jaromanda X, Jun 29 '18 at 02:29
thats the thing im not able to figure out , it was working fine a few days back . After that they changed thier site and renewed it . After that its not working in nodejs. :/ — Natesh bhat, Jun 29 '18 at 02:46

score 0 · Answer 1 · answered Dec 26 '22 at 06:58

I know this is old, but I had a similar issue. node-fetch worked on github, but not on another site I was wanting to access. wget worked fine, however. Using netcat, I analyzed the differences in headers between the fetch request vs wget's request. The difference was that fetch would request with the connection=close while wget would request with connection=keep-alive. Adding this to my fetch header got the job done.

{
  "Connection": "Keep-Alive"
}

See this post for more info on using netcat. Made it super easy to debug and I didn't previously know about this tool.

why doesn't node-fetch work on this site?

1 Answers1