4

Is there anyway to filter the log streams with patterns using the CloudWatch console?

For example, I have the following log streams in a log group - Log Group:

'/var/prod/logs'.

Log Streams:

/prod/[node_instance_id]/nginx_access
/prod/[node_instance_id]/nginx_error

I have a multi-node environment with auto scaling etc. So the log streams can be quite messy - here is an example of what I see in the log streams.

/prod/1a2b3c4d5e/nginx_access
/prod/1a2b3c4d5e/nginx_error
/prod/1b2b3c4d5e/nginx_access
/prod/1b2b3c4d5e/nginx_error
/prod/1c2b3c4d5e/nginx_access
/prod/1c2b3c4d5e/nginx_error

I am trying to filter the log streams to be all 'nginx_access' only. But from the console, it looks like I can only specify the prefix. Is there anyway that I could filter log streams using something like '*nginx_access'?

Anthony Neace
  • 25,013
  • 7
  • 114
  • 129
Ray T
  • 193
  • 2
  • 10

1 Answers1

3

The DescribeLogStreams API only supports filtering by prefix, and the console is listing your log streams with that API. It is not possible to filter by something other than a prefix.

The best practice in your case would be to use 2 log groups, one for each type:

  • /var/prod/nginx_access
  • /var/prod/nginx_error

That way you can navigate to your logs by first choosing the appropriate log group, and then searching for the instance id with the prefix filter.

Daniel Vassallo
  • 337,827
  • 72
  • 505
  • 443
  • Thanks for the info Daniel! It's useful to know that the console is actually making used of this API calls. Just out of curiosity, This [FilterLogEvents API](http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html) seems to be what I am looking for. Do you know if the console is making used of this API at all? – Ray T Apr 10 '17 at 23:52
  • 1
    @RayT - Not when listing log streams. The console uses DescribeLogStreams for that. FilterLogEvents is used for searching within the log streams or log groups. – Daniel Vassallo May 02 '17 at 15:57
  • I'm wondering how can I actually search by "prefix" from CloudWatch console. I'm only able to search the whole log group like this: https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logEventViewer:group=/aws/batch/job; or the particular stream: https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logEventViewer:group=/aws/batch/job;stream=/my-app/... But I need to use the whole stream name - I'd like to use just a prefix like "/my-app/*" – Juraj Martinka Apr 15 '20 at 07:24