centralizing logging from many locations

Question

I'm trying to find best practice to centralizing logging from many locations -server, android devices.

Current my logging condition is as below.

1. log4net - ASP.NET IIS

   • it saves warning, error log to it's IIS Server local storage.
   • When it needed, connect to each server and investigate log files.

2. slf4/log4j, FireCrash - Android app

   • it saves log cats log to its local storage
   • When it needed, my app zips it and send it by 'Android share' feature.
   • only runtime exception, it alert to me by the FireCrash solution.

3. log4js - Node JS

   • it saves my application's log to it's local storage, and connect to the server when it needed.

1st time, I tried to find a solution to gathering it's log to AWS S3. I found an s3-specific appender for log4net. But not at slf4j and log4js.

2nd time, I tried to use AWS cloud watch appender(https://github.com/camitz/CloudWatchAppender). It worked in my log4net environment. But it is not easy monitoring and not for my android, nodejs.

Someone recommended Elastic stack(ElasticSearch, Logstack, Kibana). But I'm not sure it might fit my cases.

How can I save all logs to one centralized storage?

Answer 1

It sounds like you're looking for a log aggregation service. There are many options, either hosted or self-hosted. The beauty with how many of these work is via local daemons that are used to ship your logs to the aggregator asynchronously.

Some self-hosted options -

• ELK - typically uses Beats for shipping https://www.elastic.co/products/beats
• Prometheus - https://prometheus.io/

Some hosted options -

• Sumo Logic - great fully hosted solution with many log collectors/shippers
• CloudWatch
• Logz.io

More can be found here - https://www.g2crowd.com/categories/application-performance-monitoring-apm