0

I am working on creating a zip of multiple files on the server and stream it to the client while creating. Initially, I was using ArchiverJs It was working fine if I was appending buffer to it but it fails when I need to add streams into it. Then after having some discussion on Github, I switched to Node zip-stream which started working fine thanks to jntesteves. But as I deploy the code on GKE k8s I Started getting Network Failed errors for huge files.

Here is my sample code :


const ZipStream = require("zip-stream");

/**
 * @summary Adding readable stream provided by https module into zipStreamer using entry method
 */
const handleEntryCB = ({ readableStream, zipStreamer, fileName, resolve }) => {
  readableStream.on("error", () => {
    console.error("Error while listening readableStream : ", error);
    resolve("done");
  });
  zipStreamer.entry(readableStream, { name: fileName }, error => {
    if (!error) {
      resolve("done");
    } else {
      console.error("Error while listening zipStream readableStream : ", error);
      resolve("done");
    }
  });
};

/**
 * @summary Handling downloading of files using native https, http and request modules
 */
const handleUrl = ({ elem, zipStreamer }) => {
  return new Promise((resolve, reject) => {
    let fileName = elem.fileName;
    const url = elem.url;
    //Used in most of the cases
    if (url.startsWith("https")) {
      https.get(url, readableStream => {
        handleEntryCB({ readableStream, zipStreamer, url, fileName, resolve, reject });
      });
    } else if (url.startsWith("http")) {
      http.get(url, readableStream => {
        handleEntryCB({ readableStream, zipStreamer, url, fileName, resolve, reject });
      });
    } else {
      const readableStream = request(url);
      handleEntryCB({ readableStream, zipStreamer, url, fileName, resolve, reject });
    }
  });
};

const downloadZipFile = async (data, resp) => {
  let { urls = [] } = data || {};
  if (!urls.length) {
    throw new Error("URLs are mandatory.");
  }
  //Output zip name
  const outputFileName = `Test items.zip`;
  console.log("Downloading using streams.");
  //Initialize zip-stream instance
  const zipStreamer = new ZipStream();
  //Set headers to response
  resp.writeHead(200, {
    "Content-Type": "application/zip",
    "Content-Disposition": `attachment; filename="${outputFileName}"`,
    "Access-Control-Allow-Origin": "*",
    "Access-Control-Allow-Methods": "GET, POST, OPTIONS"
  });
  //piping zipStreamer to the resp so that client starts getting response
  //as soon as first chunk is added to the zipStreamer
  zipStreamer.pipe(resp);
  for (const elem of urls) {
    await handleUrl({ elem, zipStreamer });
  }
  zipStreamer.finish();
};

app.post(restPrefix + "/downloadFIle", (req, resp) => {
  try {
    const { data } = req.body || {};
    downloadZipFile(data, resp);
  } catch (error) {
    console.error("[FileBundler] unknown error : ", error);
    if (resp.headersSent) {
      resp.end("Unknown error while archiving.");
    } else {
      resp.status(500).end("Unknown error while archiving.");
    }
  }
});

I tested for 7-8 files of ~4.5 GB each on local, it works fine and when I tried the same on google k8s, I got network failed error. After some more research, I Increased server timeout on k8s t0 3000 seconds, than it starts working fine, but I guess the increasing timeout is not good. Is there anything I am missing on code level or can you suggest some good GKE deployment configuration for a server that can download large files with many concurrent users? I am stuck on this for the past 1.5+ months. please help!

Edit 1: I edited the timeout in the ingress i.e Network services-> Load Balancing ->edit the timeout in the service Backend Service Detail_1

Raghu Chahar
  • 1,637
  • 2
  • 15
  • 33
  • How long does it take to move these large zip files through your network? Your timeout setting must be longer than that time. – O. Jones Feb 29 '20 at 13:44
  • @O.Jones Downloading gets started in 2-5 seconds depending on network. Problem is that I am no able to figure out any pattern in network failed error i.e On some networks I am getting the error after downloading 150 MB and on others, it is after 1.5-2.5 GB. Also, time after which I got this error is also not fixed, sometimes error occurs before 5 minutes and sometimes it is after 1-2 hours also. – Raghu Chahar Feb 29 '20 at 13:56
  • @RaghuChahar, where exactly have you set this timeout ? – mario Mar 02 '20 at 16:44
  • @mario I edited the question, please have a look. – Raghu Chahar Mar 03 '20 at 10:04
  • @Raghu Chahar, to be honest I have no other idea how you can do it the other way without necessity of increasing timeout to some considerably bigger value. It looks like [reasons for unexplained connection timeouts in kubernetes](https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02) can be even more complicated. – mario Mar 10 '20 at 12:44

0 Answers0