2

I am currently writing a website for an embeded system. The objective is to read a log file and modifiy a json file with the log file content.

The log file is being constantly written to by another program

The web application will be done with NodeJs and socket.io. Maybe is there another way to display those log data in real time on the html page ?

If I just open the log file, will it lock it ? Will I have the time to open it, do the algorithm and close it in 1 second ?

Is there a way to keep it open and always read the last line ?

I found this post : "Reading from a frequently updated file" but it is with Python.

Additonal info The program which writes in the log file can be updated. One of the possibility is to edit a file with a single line (which is the last one from the big log file). This unique line will be updated every second

Community
  • 1
  • 1
Weedoze
  • 13,683
  • 1
  • 33
  • 63
  • You are looking for something like Linux's **tail** command, right? Take into account that, as the log file grows, reloading it will take increasing time (consider that your approach appears to include a case where the log file size may reach several MB or more... not practical). – FDavidov Jun 22 '16 at 12:59
  • Yes sir, tail but I only need the last line which is a new one every second – Weedoze Jun 22 '16 at 13:11
  • Hummm.... There may be solutions (like the one suggested by Carlos below, but... How big could the log file become? See? getting the "TAIL" of a file might mean actual scanning of it to find the last record. If you have any control on the program that writes the log, I would suggest that this program writes twice: One time appends a line to the log while the second time writes into another file the last written line (replacing the previous contents). This might look a little bit ugly, but in terms of performance it will perhaps be the best solution. – FDavidov Jun 22 '16 at 13:22
  • Take a look at my edit :) – Weedoze Jun 22 '16 at 13:27
  • That's right. Go for it. – FDavidov Jun 22 '16 at 13:28
  • And then ? Should I still use the node-tail proposed by Carlos ? – Weedoze Jun 22 '16 at 13:30
  • Not in my opinion. If what you want is to see what is written into the log file FROM THE MOMENT you start monitoring (i.e. history is not interesting), the only thing you need to do is to read the SMALL file once/sec and append whatever is there to your JSON till you exit the monitoring mode. – FDavidov Jun 22 '16 at 13:32
  • If, on the other hand, you need to include history, load the whole log file once and then monitor the small file (take into account that generating the JSON for large amount of data will take quite some time, certainly more than one second). – FDavidov Jun 22 '16 at 13:34
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/115311/discussion-between-weedoze-and-fdavidov). – Weedoze Jun 22 '16 at 13:37

1 Answers1

4

If you're going to use Node.js I recommend you use node-tail.

It allows you to "tail" the file in a straight forward manner. According to the documentation this is how you'd do it:

Tail = require('tail').Tail;

tail = new Tail("fileToTail");

tail.on("line", function(data) {
    console.log(data);
});

tail.on("error", function(error) {
    console.log('ERROR: ', error);
});

Also, you could use fs.watchFile which comes with Node.js.

Carlos Perea
  • 596
  • 4
  • 9
  • Thx for your answer. I am not familiar with node-tail. Does the "tail.on("line"..." will be called every second when the log is updated with a new line ? – Weedoze Jun 22 '16 at 13:15
  • Tail will simulate a "tail -f" command which outputs the appended data as the file grows. It's not called every second but rather when the file grows. The event is asynchronous. – Carlos Perea Jun 22 '16 at 13:24
  • Following my edit, should I still use node-tail for 1 line ? – Weedoze Jun 22 '16 at 13:32
  • If you have access to the program that writes to the log file, then there's no need to tail the last line of the log file. You could simply take the data that will be written and do whatever you wish with it at the very moment that it will be written on the log file. My proposed solution works when you want to monitor files without modifying the actual program that writes to the file. – Carlos Perea Jun 22 '16 at 13:42
  • So I should just use a open(file) with an interval of 1sec ? – Weedoze Jun 22 '16 at 13:44