fs.createWriteStream is streaming but not writing

Question

I only recently noticed that the following code is not doing what it was doing earlier… actually writing the file to the disk.

const download = async (archive) => {
    const d = await new Promise((resolve) => {      
        const opts = { method: 'GET' };
        const req = https.request(url, opts, (res) => {
            const file = fs.createWriteStream(`path/to/${archive}`);

            res.on('data', (chunk) => {
                    file.write(chunk);
                    //process.stdout.write(chunk);
                })
                .on('end', () => file.end());
    
            resolve(archive);
        });
        
        req.on('error', (error) => console.error(error));
        req.end();
    });
}

the file is getting created on disk, but is only 0B. If I uncomment process.stdout.write(chunk), it happily spits out gobbledegook of the binary file to the console. But nothing is actually written to the disk.

Where am I goofing up? or has something changed in node recently that is silently screwing up my code? Very happy for any guidance.

This general code works for me in node v18.19.0. I don't see anything here that would have changed in versions of nodejs. One problem I see is that you are resolving the promise LONG before the stream is done. You should put the `resolve(archive)` into the `end` event handler because things are not done until then. Also, you have no `res.on('error', ...)` error handler. Another problem I see is that you don't have any flow control on the `file.write()` so for very big files your stream may not be able to keep up with the network. — jfriend00, May 13 '23 at 22:56

punkish · Accepted Answer · 2023-05-14T17:19:36.113

update: I spoke to soon. See comment below
update2: this helped me. The full zip file is downloading and unzipping properly now

thanks to the comment from @jfriend00, the following edit fixed the problem

const download = async (archive) => {
    const d = await new Promise((resolve) => {      
        const opts = { method: 'GET' };
        const req = https.request(url, opts, (res) => {
            const file = fs.createWriteStream(`path/to/${archive}`);

            res.on('data', (chunk) => {
                    file.write(chunk);
                })
                .on('end', () => {
                    file.end());
                    // moving resolve() here
                    resolve(archive);
                });
    
            //resolve(archive);
        });
        
        req.on('error', (error) => console.error(error));
        req.end();
    });
}

I could have sworn the original code was working fine and then stopped working, but I am glad I didn't swear.

update:

The edits above do not help. While now the file does get downloaded, it is not downloaded properly. It is a zip file, but when I try to unzip it, I get an error that

Error: Command failed: … 
  End-of-central-directory signature not found.  Either this 
  file is not a zipfile, or it constitutes one disk of a 
  multi-part archive.  In the latter case the central directory 
  and zipfile comment will be found on the last disk(s) of this 
  archive.

But when I download the file directly, it unzips just fine. So my code is not downloading the file properly.

fs.createWriteStream is streaming but not writing

1 Answers1