0

I am trying to push the cloudwatch logs to elastic search either using a Lambda function or Amazon Kinesis. I have the log groups setup and the elastic search domain running using terraform. Please suggest on how can I push the logs from the log group to elastic search. Please share if you have the terraform codes for the same.

Bond
  • 855
  • 3
  • 13
  • 24
  • Please specify what have you tried already, what's working, what's not, etc. to expect an answer. – Manish Kumar Sharma May 10 '17 at 21:13
  • Thanks for responding. I am looking for guidance on how/what is best. I was able to do manually "stream to amazon elastic search" from the console and in the interim setting up the IAM role (default) and also Amazon provides the default Lambda function as well. With that I am pushing the logs to elastic search. Now, I am not sure how/which terraform modules should be used to set it up. – Bond May 10 '17 at 21:23
  • I have a module written to handle this; however it's closed-source. I'm looking to get this approved to open-source it soon and will follow-up with a link when I do. – TJ Biddle Nov 29 '17 at 05:03
  • Thanks @TJBiddle – Bond Nov 29 '17 at 06:58

1 Answers1

1

This answer documents some example Terraform code for creating a lambda and Cloudwatch subscription that ships logs from a Cloudwatch log group to a Sumologic HTTP collector (just a basic HTTP POST endpoint). The Cloudwatch subscription invokes the Lambda every time a new batch of log entries is posted to the log group.

The cloudwatch-sumologic-lambda referred to in that Terraform code was patterned off of the Sumologic Lambda example.

I'd imagine you would to do something similar, but re-writing the Lambda to format the HTTP however ElasticSearch requires. I'd bet some quick googling on your part will turn up plenty of examples.

Alternatively to all this Terraform config though, you can just go to your Cloudwatch console, select the log group you're interested in and select "Stream to Amazon ElasticSearch".

enter image description here

Though I think that will only work if you're using the AWS "ElasticSearch service" offering - meaning if you installed/configured ElasticSearch on some EC2 instances yourself it probably won't work.

Community
  • 1
  • 1
Shorn
  • 19,077
  • 15
  • 90
  • 168
  • Thanks for your inputs Shorn. I did the "Stream to Amazon Elasticsearch service" manually and it works fine. I am using Amazon Elastic search. They made it simple and automates the creation of the lambda function as well. May be I could just reuse the same lamda function that Amazon provides by default. I was confused as to what Terraform module I should be using to automate this but looks like aws_cloudwatch_log_subscription_filter and aws_lambda_function are the ones I should be using since I already have the Log group created. already. Btw, I saw the other post you responded as well. – Bond May 11 '17 at 03:44
  • I am getting the below error while trying to subscribe to Kinesis Firehose – Bond May 12 '17 at 21:46
  • * aws_cloudwatch_log_subscription_filter.test_kinesis_logfilter: 1 error(s) occurred: * aws_cloudwatch_log_subscription_filter.test_kinesis_logfilter: InvalidParameterException: Could not deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE state. – Bond May 12 '17 at 21:52