4

I'd like to do something like:

var fs = require('fs');
var through = require('through');

var file = 'path/to/file.json';

var input = fs.createReadStream(file, 'utf8');
var output = fs.createWriteStream(file, 'utf8');

var buf = '';
input
.pipe(through(function data(chunk) { buf += chunk; }, function end() {
  var data = JSON.parse(buf);
  // Do some transformation on the obj, and then...
  this.queue(JSON.stringify(data, null, ' '));
})
.pipe(output);

But this fails because it's trying to read and write to the same destination. There are ways around it, like only piping to output from within the end callback above.

Is there a better way? By better, I mean uses less code or less memory. And yes, I'm aware that I could just do:

var fs = require('fs');
var file = 'path/to/file.json';

var str = fs.readFileSync(file, 'utf8');
var data = JSON.parse(str);    
// Do some transformation on the obj, and then...
fs.writeFileSync(file, JSON.stringify(data, null, '  '), 'utf8');
Andrew
  • 3,332
  • 4
  • 31
  • 37

1 Answers1

1

There is no other way that your code will use less memory, because you need the whole file to parse it into a Javascript object. In this way, both versions of your code are equivalent memory-wise. If you can do some work without having to work on the full JSON object, check out JSONStream.

In your example, you should read the file, then parse and transform it, then write the result to a file; although you shouldn't use the synchronous version of the functions, see the end of this paragraph of the Node.js documentation:

In busy processes, the programmer is strongly encouraged to use the asynchronous versions of these calls. The synchronous versions will block the entire process until they complete--halting all connections.

Anyway, I don't think you can read from a file while you're overwriting it. See this particular answer to the same problem.

Community
  • 1
  • 1
Paul Mougel
  • 16,728
  • 6
  • 57
  • 64
  • While I agree that the synchronous versions should be avoided, in this example they resulted in drastically less code for a quick and temporary build step (not a long-running server process). Thanks for the link to the bash answer, it was helpful! – Andrew Nov 15 '13 at 20:28
  • Indeed. Could you show us the content of the input file and the kind of transformation you wish to achieve? Without this, it's hard to know if the algorithm is streamable or not. – Paul Mougel Nov 15 '13 at 20:30
  • The content was a really simple JSON file, `{ "key": "value", "someKeyToDelete": { "yep": "i won't be there" } }`, where I was just deleting `someKeyToDelete` from it. – Andrew Nov 16 '13 at 00:30
  • I've found that I can use streams to overwrite a file I'm reading, but not reliably--sometimes it will work fine, and other times I end up with an empty file. Better to write to a temporary file, and then delete the original and rename the temp file over it. – DawnPaladin Jan 30 '15 at 15:37