How does one get the html of a webpage from a url

Question

So this question I thought would be beyond simple. Yet after a solid hour of searching JavaScript and node js documentation and other questions on this site I've only found things that have to do with the same origin policy.

What I want is to make a simple request to a url and read it's html/source code. This is to query a wiki and return the top link from the search results. I can handle dealing with the html myself, but I am absolutely dumbfounded as to how to get the html. I know how to do it very easily with Lua socket in a single line, which is how I did this when making a twitch bot, but now that I am adapting it to discord I'm coding it in JavaScript, and I am not as well versed in JavaScript.

Does anybody know how to go about this?

My current code:

function SearchWiki(query) {
    console.log("I was called, I guess")
    var options = {
        path: encodeURI(query),
        method: "GET"
    }
    var request = http.request("http://pm-challenge.wikia.com/wiki/Special:Search?search=", options, function (res) {
        var data = '';
        res.on('data', function (chunk) {
            data += chunk;
        });
        res.on('end', function () {
            console.log(data);  
        });
    });
    request.on('error', function (e) {
        console.log(e.message);
    });
    request.end();
}

https://stackoverflow.com/questions/6287297/reading-content-from-url-with-node-js — Adam H, Sep 05 '18 at 19:34
If you mean to be able to request it from a browser then it will really depend on your requirements. In modern chrome you can use `fetch` with `no-cors` options. If you have to support many types of browsers then the best option is to create an end point to proxy the request. Then you can have it return whatever you want and not have to deal with browsers. — Austio, Sep 05 '18 at 19:37
Looking into these links, thanks guys, will post back in a bit — warspyking, Sep 05 '18 at 19:37
@Austio — The question is tagged [tag:node.js] and browsers are not mentioned. Even if the question was about browsers, if the `no-cors` option was used, then he wouldn't be able to read the response! That's the point of `no-cors` (to silently fail at things CORS permission is required for instead of throwing an error). — Quentin, Sep 05 '18 at 19:39
@AdamH When I use the code in the answer to that question I get the following output: `Moved to http://pm-challenge.wikia.com/wiki/Special:Search?search=` — warspyking, Sep 05 '18 at 19:41
Sounds like you got a 301 (moved) code, try using the URL they suggest. — Adam H, Sep 05 '18 at 19:42
@AdamH Oh, I see. Alright I changed my url and just made the path in the options the search query itself. My issue now is a 400 bad request. How do I fix that? — warspyking, Sep 05 '18 at 19:46
@warspyking — We have no way of knowing what request you are sending or what the server you are making the request to considers bad about it. — Quentin, Sep 05 '18 at 19:46
Take a look here to see what you are doing wrong with the options and passing the URL: https://nodejs.org/api/http.html#http_http_request_options_callback — Adam H, Sep 05 '18 at 20:30
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/179495/discussion-between-warspyking-and-adam-h). — warspyking, Sep 05 '18 at 20:33
the request callback function takes 3 params. The thrid param is the HTML body. https://www.npmjs.com/package/request — Astrydax, Sep 13 '18 at 16:49

How does one get the html of a webpage from a url

0 Answers0