Edit large file in node

Question

I have a file test.txt with unknown size. Shared file with other services and i have to read from this file to edit it. Just small edit changing timestamp to now. What is the best way to edit it without read the whole file and rewrite again. I don't feel that this a proper way to do it. I am aware of createReadStream and createWriteStream but i don't want copy the file and waste resources especially the memory. Thanks.

Answer 1

I don't know of a way where you can read the contents of a file to change without at least opening the file, changing what you need to change and then re-writing it. The most efficient and performant way in Node to do this would be through streams since you don't need to read the entire file all at once. Assuming your file you need to edit has a new line or carriage return you can use the Readline module to iterate the file in question line-by-line and check to see if the line contains the text you'd like to change. Then you can write that data to the file in place where the old text was.

If you don't have line breaks you could opt to use a Transform Stream and check each chunk for the matching text but this might require stitching together multiple chunks to identify the text to replace.

I know you don't want to more or less copy the file with the changes made, but I couldn't come up with another approach that would be as efficient.

const fs = require('fs')
const readline = require('readline')

const outputFile = fs.createWriteStream('./output-file.txt')
const rl = readline.createInterface({
    input: fs.createReadStream('./input-file.txt')
})

// Handle any error that occurs on the write stream
outputFile.on('err', err => {
    // handle error
    console.log(err)
})

// Once done writing, rename the output to be the input file name
outputFile.on('close', () => { 
    console.log('done writing')

    fs.rename('./output-file.txt', './input-file.txt', err => {
        if (err) {
          // handle error
          console.log(err)
        } else {
          console.log('renamed file')
        }
    }) 
})

// Read the file and replace any text that matches
rl.on('line', line => {
    let text = line
    // Do some evaluation to determine if the text matches 
    if (text.includes('replace this text')) {
        // Replace current line text with new text
        text = 'the text has been replaced'
    }
    // write text to the output file stream with new line character
    outputFile.write(`${text}\n`)
})

// Done reading the input, call end() on the write stream
rl.on('close', () => {
    outputFile.end()
})

Answer 2

If you just want to change timestamps, you can use fs.futimes(). Is native to node up from version v0.4.2.

var fs = require("fs");

var fd = fs.openSync("file"); // Open a file descriptor

var now = Date.now() / 1000;
fs.futimesSync(fd, now, now); // Modify it by (fd, access_time, modify_time)

fs.closeSync(fd); // Close file descriptor

This way you don't depend on any npm package.

You can read more here: https://nodejs.org/api/fs.html#fs_fs_futimes_fd_atime_mtime_callback

Answer 3

You need something like touch linux command line and there is a npm package doing exactly this.