Why does this GET request seem to give random characters?

Question

This is 1 of my first times using the https node.js module and I tried to make a GET request with this code

const url = new URL("valid url") //I did put a valid url

https.get({
    host: url.hostname,
    path: url.pathname,
    headers: {
        "Content-Type": "application/json"
    }
}, (res) => {
    res.setEncoding("utf8")
    let str = ""
    res.on("data", c => {
        console.log(c)
    })

    res.on("end", () => {
        console.log(str) //this was meant to log the data but I removed the "str += c" from the data callback
    })
})

But this logs the following:

▼�
�VJ-*�/��LQ�210ЁrsS��‼�S����3KR§2�§�R♂K3�RS�`J�↕sA�I�)�♣�E@NIj�R-��♥g��PP

But I would think .setEncoding("utf8") would work. Is it something to do with the headers? Or maybe the URL object? The result I want is in a <pre> element and is valid JSON.

Can you show what the same request does with `curl` ? what does it actually return. Data is rarely ever random. It's probably some binary but hard to say with what you've shared so far. — Evert, Nov 08 '21 at 19:50
Either way - for all we know, your `valid url` is a random string generator. Also, you'll want to look at the response headers. If it says `content-encoding: gzip`, it's compressed... — AKX, Nov 08 '21 at 19:50
@MrMythical no, it's a popular CLI tool to do HTTP requests. It will let you make a simple request and compare results. — Evert, Nov 08 '21 at 19:51
This code logs what I want (a JSON object), but I want to see how to do it with the https package: `fetch(url).then(async val => { console.log( await ( val.json() ) ) })` and `fetch` is the `node-fetch` package — MrMythical, Nov 08 '21 at 19:55
The content looks compressed. I also suspect gzip. [This might help](https://stackoverflow.com/questions/10207762/how-to-use-request-or-http-module-to-read-gzip-page-into-a-string) — James, Nov 08 '21 at 20:00

MrMythical · Accepted Answer · 2022-03-11T16:28:05.300

As said by @James in their comment, the content was compressed. Doing this worked:

import https from "https"

import { createGunzip } from "zlib"

const url = new URL("valid url")

const req = https.get(url, (res) => { //I found out you can use a URL object
    
    let str = ""
    let output;
        //See if it's compressed so we can read it properly
        var gzip = createGunzip()
        res.headers["content-encoding"] == "gzip" ? 
        output = res.pipe(gzip) :
        output = res;

    output.on("data", c => {
        str = c.toString()
    })

    output.on("end", () => {
        console.log(JSON.parse(str || "{}")) //logs the correct object
    })
})

Why does this GET request seem to give random characters?

1 Answers1