3

Looking for a little help getting started on a little project i've had in the back of my mind for a while.

I have log file(s) varying in size depending on how often they are cleaned from 50-500MB. I'd like to write a program that will monitor the log file while its actively being written to. when in use it's being changed pretty quickly easily several hundred lines a second or so. Most if not all of the examples i've seen for reading log/text files are simply open and read file contents into a variable which isn't really feasible to do every time the file changes in this situation. I've not settled on a language to write this in but its on a windows box and I can work in .net flavors / java / or php ( heh dont think php will fly to well for this), and can likely muddle through another language if someone has a suggestion for something well built for handling this.

Essentially I believe what I'm looking for would probably be better described to as a high speed way of monitoring a text file for changes and seeing what those changes are. Each line written is relatively small. (less than 300 characters, so its not big data on each line).

EDIT: to change the wording to hopefully better describe what i'm trying to do. Which is write a program to keep an eye on a log file for a trigger then match a following action to that trigger. So my question here is pertaining to file handling inside a programming language.

I greatly appreciate any thoughts/comments.

Nathan
  • 1,445
  • 2
  • 11
  • 29
  • 1
    Have you considered a tool like BareTail? http://www.baremetalsoft.com/baretail/ – Mark Thomas Oct 18 '10 at 02:31
  • I like the looks of that for other uses, but for this I need to act on a trigger in the log file and then monitor for an action that should follow. – Nathan Oct 21 '10 at 16:29

3 Answers3

0

The 'keep an eye on a log file' part of what you are describing is what tail does.

If you plan to implement it in Java, you can check this question: Java IO implementation of unix/linux "tail -f" and add your trigger logic to lines read.

Community
  • 1
  • 1
fglez
  • 8,422
  • 4
  • 47
  • 78
0

If it's incremental then you can just read the whole file the first time you start analyzing logs, then you keep the current size as n. Next time you check (maybe a timed action to check last modified date) just skip first n bytes, read all new bytes and update size.

Otherwise you could use tail -f by getting its stdout and using it for your purposes..

Jack
  • 131,802
  • 30
  • 241
  • 343
0

I suggest not reinventing the wheel. Try using the elastic.co

enter image description here

All of these applications are open source and free and are capable of monitoring (together) and trigger actions based on input.

  • filebeats - will read the log file line by line (supports multiline log messages as well) and will send it across to logstash. There are loads of other shippers you can use.
  • logstash - will take the log messages, filter them, add tags and send the messages to elasticsearch
  • elasticsearch - will take the log messages and index them, the store them. It is also capable of running actions based on input
  • kibana - is a user friendly web interface to query and analyze the data. Or just simply put it up on a dashboard.

    Hope this helps.

    • Viorel Florian
    • 594
    • 9
    • 29