using https.get instead of got causes a 308

Question

this is a difficult question to ask because I am mystified, but let's see…

I am comparing Got with https.get, and have the following, bare simple code that works. Both Got and https.get return exactly the same result.

But when I use exactly the same code in my Fastify application, Got works as expected but https.get results in a 308.

Is there some way I can debug this code to see what is being sent out by https.get that is causing the remote server to respond with a 308 instead of 200?

import got from 'got';
import https from 'https';

const withGot = async (uri) => {
    try {
        const json = JSON.parse((await got(uri)).body);
        console.log(json);
    }
    catch (error) {
        console.error(error);
    }
}

const withHttps = async (uri) => {
    try {
        const json = await getRequest(uri);
        console.log(json);
    }
    catch (error) {
        console.error(error);
    }
}

const getRequest = async (uri) => {
    
    return new Promise((resolve) => {
        https.get(uri, (res) => {
            const { statusCode } = res;
            const contentType = res.headers['content-type'];
            
            let error;

            /**
             * Any 2xx status code signals a successful response but
             * here we're only checking for 200.
            **/
            if (statusCode !== 200) {
                error = new Error(`ERROR\n${'-'.repeat(50)}\nRequest Failed.\nURI: ${uri}\nStatus Code: ${statusCode}`);
            } 
            else if (!/^application\/json/.test(contentType)) {
                error = new Error(`Invalid content-type.\nExpected application/json but received ${contentType}`);
            }

            if (error) {
                console.error(error.message);

                /**
                 * Consume response data to free up memory
                **/
                res.resume();
                return;
            }

            res.setEncoding('utf8');
            let rawData = '';
            res.on('data', (chunk) => { rawData += chunk; });
            res.on('end', () => {
                try {
                    const parsedData = JSON.parse(rawData);
                    resolve(parsedData);
                } 
                catch (e) {
                    console.error(e.message);
                }
            });
        }).on('error', (e) => {
            console.error(`Got error: ${e.message}`);
        });
    });
}

const uri = 'https://zenodo.org/api/records/?q=phylogeny';

withGot(uri);
withHttps(uri);

If 2 HTTP clients get different results for the same request, something must be different between the two of them. Your first step should be to exactly figure out what is sent to the server. Most likely there's a difference in the headers, but try to enumerate this whole list. One way to do this is using a debug proxy, or by hitting a small test server. — Evert, Jul 23 '22 at 20:33
Also is it possible that *both* get the 308 but the `got` version follows the redirect automatically? — Evert, Jul 23 '22 at 20:36
@evert, good points, both of them, but keep in mind, both **Got** and **https.get** work exactly the same in the standalone script. They produce different results only when embedded inside my Fastify app (even though the embedded code is exactly the same as the standalone script). So, something about being in the bigger program is causing this and I want to find out what. Makse sense something different is being sent to the server, but what? How do I set up a debug proxy? How do I intercept what **https.get** is sending when in the bigger program? — punkish, Jul 23 '22 at 21:43
ok, I made a bit of progress… if I replace `https.get` with `https.request`, it works even when embedded within my larger program. I have to now go through line by line and see if there is something that is causing `https.get` to fail — punkish, Jul 23 '22 at 21:57
ok, I figured it out, I have posted an update in the original post — punkish, Jul 23 '22 at 22:04
Glad you figured it out! and thank you for the update. It's always welcomed even if my comments weren't helpful in the end — Evert, Jul 24 '22 at 04:14

punkish · Answer 1 · 2022-07-23T22:25:47.377

I figured out the reason for the problem (and the solution)… seems like when I use https.get, I still have to pass the options with a port 443 (the default port for https), otherwise, https seems to knock on port 80 and then gets redirected to port 443 which results in the server sending back html which causes the JSON parser to croak. If I pass an options object like below, then it works. But, it is still weird that the standalone script works fine without the options, so I continue to be mystified even though I have found a solution.

const options = {
    hostname: 'zenodo.org',
    port: 443,
    path: `/api/records/?${qs}`,
    method: 'GET',
    headers: {
        'Content-Type': 'application/json'
    }
};

using https.get instead of got causes a 308

1 Answers1