1

We have a cloud application running on EC2 instances. At present we are running 20 instances but in future it may go up to 50 instances. Each instance generates about 100 MB of logs per day. We want the logs from all these instances to be available at a central location. When a new instance is added to monitoring, a separate folder gets created on the central log server which will store the log files generated on this instance. Inside the instance's folder, log files are replicated exactly as it was on the instance where it is generated.

For ex:
On instance-1 say I have asked to monitor files /home/app/binary/*.log. Now when a file /home/app/binary/a.log is created/updated, the same file should be automatically created/updated to the central server folder instance-1/binary/a.log

On each instance we will monitor a lot of different log files including my application logs, 3rd party logs, system logs etc.

Please suggest a tool to handle this requirement.

Other required features are:

  1. I should be able to monitor directories and also use wildchars in path.
  2. Logs should be replicated to central server in near real time.
  3. The tool must ensure reliable delivery of logs.
  4. As I explained above, the central log server should store logs in files with the same folder-name/file-name as in source.

Edit: I would prefer a stable tool under active development with support available.

Varun
  • 4,054
  • 6
  • 31
  • 54

1 Answers1

0

Have a look at these similar questions already answered on SO: Aggregate multiple log files

Manage large number of log files

Community
  • 1
  • 1
stamhaney
  • 1,246
  • 9
  • 18
  • Already gone through the links you provided. None of the mentioned approaches meets all my requirements. Hence posted again. – Varun Dec 28 '12 at 07:29