Best way to aggregate multiple log files from several servers

Question

I need a simple way to monitor multiple text log files distributed over a number of HP-UX servers. They are a mix of text and XML log files from several distributed legacy systems. Currently we just ssh to the servers and use tail -f and grep, but that doesn't scale when you have many logs to keep track of.

Since the logs are in different formats and just files in folders (automatically rotated when they reach a certain size) I need to both collect them remotely and parse each one differently.

My initial thought was to make a simple daemon process that I can run on each server using a custom file reader for each file type to parse it into a common format that can be exported over the network via a socket. Another viewer program running locally will connect to these sockets and show the parsed logs in some simple tabbed GUI or aggregated to a console.

What log format should I try to convert to if I am to implement it this way?

Is there some other easier way? Should I attempt to translate the log files to the log4j format to use with Chainsaw or are there better log viewers that can connect to remote sockets? Could I use BareTail as suggested in another log question? This is not a massivly distributed system and changing the current logging implementations for all applications to use UDP broadcast or put messages on a JMS queue is not an option.

In the application I'm working at now we've solved this problem by sharing the interesting parts of the logs (exceptions, server load and user stats) and showing them on an internal statuspage for admin users. Works way better than following noisy log files. — Claes Mogren, Mar 29 '11 at 10:38
See http://serverfault.com/questions/180392/how-to-collect-and-aggregate-logs-from-multiple-machines-in-amazon-aws-environme — ripper234, Jan 18 '12 at 17:21
Have a look at [logstash](https://docs.google.com/present/view?id=dcmwwd94_16dfdxgpw8&pli=1)? It has loads of input, output and filter options. — Varun, May 21 '12 at 09:10

score 23 · Answer 1 · answered Aug 13 '10 at 17:36

23

Probably the lightest-weight solution for real-time log watching is to use Dancer's shell in concurrent mode with tail -f:

dsh -Mac -- tail -f /var/log/apache/*.log

The -a is for all machine names that you've defined in ~/.dsh/machines.list
The -c is for concurrent running of tail
The -M prepends the hostname to every line of output.

answered Aug 13 '10 at 17:36

mrm

5,001
2
32
30

2

sponsered by  Computers – coderatchet Aug 25 '16 at 22:45
This is great thanks! dsh is able to use `~/.ssh/config` if `remoteshell = ssh` is specified in `dsh.conf`or passed as an argument, i.e. `-r ssh`. You can also create group definitions (at least with the brew installed version in your home folder, e.g. ~/.dsh/group/prod.web) and modify the command a bit: `dsh -Mc -g prod.web`. – Andrew Mackrodt Mar 07 '17 at 13:15

masukomi · Answer 2 · 2016-04-22T12:30:43.207

We use a simple shell script like the one below. You'd, obviously, have to tweak it somewhat to tell it about the different file names and decide which box to look for which on but you get the basic idea. In our case we are tailing a file at the same location on multiple boxes. This requires ssh authentication via stored keys instead of typing in passwords.

#!/bin/bash
FILE=$1
for box in box1.foo.com box2.foo.com box3.foo.com box4.foo.com; do
     ssh $box tail -f $FILE &
done

Regarding Mike Funk's comment about not being able to kill the tailing with ^C, I store the above in a file called multitails.sh and appended the following to the end of it. This creates a kill_multitails.sh file which you run when you're done tailing, and then it deletes itself.

# create a bash script to kill off 
# all the tails when you're done
# run kill_multitails.sh when you're finished

echo '#!/bin/sh' > kill_multitails.sh
chmod 755 kill_multitails.sh
echo "$(ps -awx | grep $FILE)" > kill_multitails_ids
perl -pi -e 's/^(\d+).*/kill -9 $1/g' kill_multitails_ids
cat kill_multitails_ids >> kill_multitails.sh
echo "echo 'running ps for it'" >> kill_multitails.sh
echo "ps -awx | grep $FILE" >> kill_multitails.sh
echo "rm kill_multitails.sh" >> kill_multitails.sh
rm kill_multitails_ids


wait

score 5 · Answer 3 · answered Oct 12 '12 at 12:26

5

Logscape - like splunk without the price tag

answered Oct 12 '12 at 12:26

JzJ

119
2
1

score 2 · Answer 4 · answered Apr 09 '12 at 16:29

multitail or

"chip is a local and remote log parsing and monitoring tool for system admins and developers.
It wraps the features of swatch, tee, tail, grep, ccze, and mail into one, with some extras"

Eg.

chip -f -m0='RUN ' -s0='red' -m1='.*' -s1 user1@remote_ip1:'/var/log/log1 /var/log/log2 /var/log/log3 user2@remote_ip2:'/var/log/log1 /var/log/log2 /var/log/log3’' | egrep "RUN |==> /"

This will highlight in red the occurences of the -m0 pattern, pre-filtering the 'RUN |==> /' pattern from all the log files.

score 2 · Answer 5 · answered Feb 14 '13 at 09:37

2

I wrote vsConsole for exactly this purpose - easy access to log files - and then added app monitoring and version tracking. Would like to know what you think of it. http://vs-console.appspot.com/

answered Feb 14 '13 at 09:37

prule

2,536
31
32

Hey man, is this project still active? When I go to your site and click on Downloads and Documentation, it opens a google drive share but the only file i see in there is a EULA.. – Richie086 Aug 09 '18 at 15:13
Hi, I never got any encouragement to keep the project going, so no - it hasn't been active for years. I'm not sure whats happening with the download link (the link above appears to work for me, even when I'm not logged in), but you should see a folder for vsConsole which has a downloads folder - try this one https://drive.google.com/drive/folders/0B9pO6BCz4ndtVl90NDJQdUtvNTQ - nowadays there are many options to solve this problem, eg https://papertrailapp.com/ – prule Aug 09 '18 at 23:34

score 2 · Accepted Answer · edited Jul 23 '14 at 16:47

2

Options:

Use a SocketAppender to send all logs to 1 server directly. (This could serverly hamper performance and add a single point of failure.)
Use scripts to aggregate the data. I use scp, ssh, and authentication keys to allow my scripts to get data from all servers without any login prompts.

edited Jul 23 '14 at 16:47

Aaron

55,518
11
116
132

answered Sep 18 '08 at 12:41

James A. N. Stauffer

2,649
3
22
32

score 1 · Answer 7 · answered Jan 15 '10 at 05:58

You can use the various receivers available with Chainsaw (VFSLogFilePatternReceiver to tail files over ssh, SocketReceiver, UDPReceiver, CustomSQLDBReceiver, etc) and then aggregate the logs into a single tab by changing the default tab identifier or creating a 'custom expression logpanel' by providing an expression which matches the events in the various source tabs.

score 1 · Answer 8 · answered Jun 25 '10 at 07:47

1

gltail - real-time visualization of server traffic, events and statistics with Ruby, SSH and OpenGL from multiple servers

answered Jun 25 '10 at 07:47

Albert T. Wong

602
1
6
15

glTail is cool, but I was more after an easy and efficient way to aggregate log-files to search and filter them. – Claes Mogren Jul 12 '10 at 09:52

score 1 · Answer 9 · answered Sep 17 '08 at 15:00

Awstats provides a perl script that can merge several apache log files together. This script scales well since the memory footprint is very low, logs files are never loaded in memory. I know that si not exactly what you needs, but perhaps you can start from this script and adapt it for your needs.

score 1 · Answer 10 · answered Feb 15 '12 at 13:52

1

XpoLog for Java

answered Feb 15 '12 at 13:52

tom

11
1

Best way to aggregate multiple log files from several servers

10 Answers10

Linked