2

I've created a file sharing app to use on my home network. It's kind of similar to the web interface for Google Drive or Dropbox but without any file size caps and no security.

It works perfectly when transferring small files instantly from other computers connected via LAN, but something bizarre happened while testing it on a 2GB file.

It took about four hours for the progress bar to get to 50% before I cut it off. Also, take a look at how much memory Node is using.

Node memory hog

It will start off low and build its way up to as much as 13GB, then dump and restart, several times a minute.

The file is uploaded from a browser interface using an XMLHttpRequest. This is the essential part of front-end code.

var formData = new FormData();
var file = document.getElementById("fileinput").files[0];
formData.append("file", file);

var xhr = new XMLHttpRequest();
xhr.open("post", "/fileupload", true);
xhr.send(formData);

And on the server side there's a very simple handler using express that takes req.files.file and passes it directly into fs.writeFile to save it to disk. ('req.files.file' seems to be a Buffer type according to a console.log, which apparently can be piped right onto disk).

var express = require("express");
var fileUpload = require("express-fileupload");
var app = express();
app.use(express.static("public"));
app.use(fileUpload());
var fs = require("fs");

app.post('/fileupload', function(req, res) {
    if(req.files && req.files.file){
        var file = req.files.file;
        fs.writeFile("./public/shared/" + file.name, file.data, "binary", function(err) {
            if(err) {
                res.send(err);
                return;
            }
            res.send("File uploaded successfully.");
        }); 
    }
    else{
        res.send("No file was uploaded.");
        return;
    }
});

I'm using express-fileupload to handle uploads.

Both computers are running Windows. The sending computer was using Chrome. Node version is 7.5.0.

Where am I going wrong? How do I approach this problem?

Community
  • 1
  • 1
JSideris
  • 5,101
  • 3
  • 32
  • 50

2 Answers2

1

You should probably use a writeStream instead of a writeFile for large files. When you use writeFile, the entire file has to be kept in memory, hence the huge memory consumption.

This was a decent short tutorial: http://joshondesign.com/2014/06/25/nodestreamsareawesome

And node.js has it's docs as well: https://nodejs.org/api/fs.html#fs_class_fs_writestream

Shilly
  • 8,511
  • 1
  • 18
  • 24
  • To the moment of time `writeFile` is executed, all the data is already in memory, as there is a reference to a huge `Buffer` instance. – Sergey Lapin Feb 02 '17 at 10:05
  • @SergeyLapin Is it? How do these buffers work? I haven't worked with HTTP buffers in the past, but my assumption is that it's a persistent stream from client to server until the entire buffer has been read. Doesn't explain why my memory usage is off the charts (13GB), but I'll give this a shot. – JSideris Feb 02 '17 at 18:07
  • @JSideris could you please answer my comment to original post? – Sergey Lapin Feb 02 '17 at 18:12
1

Look at express-fileupload source. When a file arrives, it takes an incoming stream and starts to put it into a Buffer, i.e. into memory. And once that is done you write this memory buffer to a disk via writeFile.

express-fileupload has no option to avoid storing a file in memory. Consider using multer with DiskStorage option it provides: docs

Sergey Lapin
  • 2,633
  • 2
  • 18
  • 20