Where can I export AWS Cloudwatch Logs (for Loggly)?

Question

I know they are storing them somewhere. Maybe a DynamoDB or S3 bucket?

I'd like to use Cloudwatch as a log archive for my EC2 instances, and set it up so that they are passed to Loggly for viewing/sorting/ etc.

I know Loggly recently released a way to grab logs from S3 buckets (https://www.loggly.com/docs/s3-logs/) but I can't seem to find the source of these logs on the AWS side.

My thoughts so far are:

• Maybe I could set up a Metric/Alarm to pass all incoming logs to Loggly?
• I could possibly gain access to where AWS stores these Cloudwatch Logs?
• Set up a cron somewhere that bounces Cloudwatch logs to an S3 bucket, and go from there?
• Use Cloudwatch API calls in a cron to directly grab logs and send them to Loggly through an HTTP post etc.

Answer 1

I've done this by setting up a subscription to the Log Group that triggers a Lambda function. The function transforms the log data and writes it as a .log file to S3 or sends the records to Kinesis Firehose for collection, and then to S3 as an archive.

You can checkout the full writeup or just the Lambda Function if you're trying to do something similar.

Answer 2

It seems that Loggly does not support similar service.

According to Loggly Log File Monitoring Options you have those options:

1. Option

   • Setup new instance with cronjob to sync log from S3 to the local disk.
   • Setup rsyslog service and send this log file to Loggly.

2. Option

   • Using Loggly File Upload how to and push log file with curl POST.

   curl -X POST -T FILE.TXT https://logs-01.loggly.com/bulk/TOKEN/tag/file_upload

Regards.

Answer 3

This API will let you export data to S3, however note it's for historical data not real time... data up to a few hours old will not be available to export, so most useful for daily extracts: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/S3Export.html

For realtime delivery you would want to use the Subscriptions API: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html

Answer 4

Some time since this question was asked, Loggly added support for Cloudwatch Logs. It uses a Lambda function, probably similar to the solution by @Michael above.

https://www.loggly.com/docs/cloudwatch-logs/