1

I am trying to append lines to a file in nodejs. I have written the following peice of code .

/* Name :  test.js */
/* globals require,__dirname */
var fs = require("fs");
for (var i = 0; i < 100000; i++) {
    fs.appendFileSync(__dirname+'/out.txt', "line " + i + "\n");
}

But when I run this program, I am getting the following error. 

$ node test.js
fs.js:584
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                                                  ^

Error: EBUSY: resource busy or locked, open 'd:\<redacted>\out.txt'
    at Error (native)
    at Object.fs.openSync (fs.js:584:18)
    at Object.fs.writeFileSync (fs.js:1224:33)
    at Object.fs.appendFileSync (fs.js:1283:6)
    at Object.<anonymous> (d:\<redacted>\test.js:3:8)
    at Module._compile (module.js:425:26)
    at Object.Module._extensions..js (module.js:432:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:313:12)
    at Function.Module.runMain (module.js:457:10)

I have searched but could not find any reason or solution for this. Can someone please explain why this Error occurs and how to fix it ?

When I try the following method it works but this is not very feasible for the problem I am attempting to solve. 

/* globals require,__dirname */
var fs = require("fs");
var buffer = "";

for (var i = 0; i < 100000; i += 1) {
    var line =  'line ' + i + '\n';
    buffer += line;                 // keep the line in buffer    
}

fs.appendFileSync(__dirname+'/out.txt', buffer);   // now write the buffer

Thanks in advance.

Subhendu Ranjan Mishra
  • 81
  • 1
  • 7
  • I think that happens because you try to access your file a lot of times in a very short period of time. What is the "problem" you are trying to solve ? – Lauromine Dec 24 '15 at 15:23
  • @Lauromine Yes, that might be the cause , but should not `fs.appendFileSync` handle this ? The problem I am trying to solve is , I have to write a program that will read a JSON dataset with > 2 million records and and split it into 3 files depending on the record. – Subhendu Ranjan Mishra Dec 24 '15 at 15:28
  • I just found http://stackoverflow.com/questions/10747347/node-js-and-file-system-on-windows-ebusy-error maybe it'll help you. – Lauromine Dec 24 '15 at 15:30
  • @Lauromine I have seen that earlier , but in that case the program was trying to access a special system file. BTW thanks for your quick response. – Subhendu Ranjan Mishra Dec 24 '15 at 15:33

1 Answers1

2

Your code works for me but it's awfully slow, it takes minutes to finish writing in the file (269621.425ms to be exact). Your issue is likely because of so many access in a very short time, I guess it works for me because I'm using an SSD.

For your issue, streams might be very helpful. It's fast and memory efficient (you don't have to store a giant buffer variable). I don't know what you want to do exactly so here is your example written using streams (I used event-stream for convenience):

var fs = require('fs');
var es = require('event-stream');

console.time('done in');
es.readable(function (count, next) {
    for (var i = 0; i < 100000; i++) {
        this.emit('data', 'line ' + i + "\n");
    }
    this.emit('end');
    console.timeEnd('done in');
    next();
}).pipe(fs.createWriteStream('out.txt'));

done in: 90.561ms

Shanoor
  • 13,344
  • 2
  • 29
  • 40