Couldn't fetch CloudWatch logs without LogStreamName specified

Question

I am working with Python 3.6 and boto3==1.7.84. I was trying to fetch CloudWatch logs with boto3 from AWS, but found that the number of events returned is much less than what I can see in the CloudWatch insights page. I supposed that

import boto3
client = boto3.client('logs')
response = client.filter_log_events(
    logGroupName='/aws/batch/job',
    startTime=1572520000000,
    endTime=1572570000000,
    filterPattern='exceptions',
)

would return all the events including 'exceptions' regardless of the job stream name. However it returned nothing. But if I specified the logStreamNames like this

import boto3
client = boto3.client('logs')
response = client.filter_log_events(
    logGroupName='/aws/batch/job',
    logStreamNames=['training/default/[ASpecificID]'],
    startTime=1572520000000,
    endTime=1572570000000,
    filterPattern='exceptions',
)

it did return the logs containing string 'exceptions' with logStreamNames=['training/default/[ASpecificID]'].

The other weird thing was that when I did

import boto3
client = boto3.client('logs')
response = client.filter_log_events(
    logGroupName='/aws/batch/job',
    logStreamNamePrefix='training/default',
    startTime=1572520000000,
    endTime=1572570000000,
    filterPattern='exceptions',
)

the logs containing string 'exceptions' with logStreamNames=['training/default/[ASpecificID]'] were not returned. Some logs with logStreamNamePrefix='training/' did show up, but not all. The number of events returned is much less than what I got by doing

fields @timestamp, @message, @logStream
| filter @logStream like /training\/default/
| filter @message like /exceptions/
| limit 10000

with CloudWatch logs insights query syntax in the CloudWatch insights page. Did I do anything wrong with boto3 that led to this discrepancy?

You need the stream name to get log events. [This answer might help you](https://stackoverflow.com/a/59254837/9499392) — Maaz Bin Mustaqeem, Jan 16 '23 at 10:42

Lamanus · Answer 1 · 2019-11-01T12:34:41.983

From the boto3 documentation, it is expected.

logStreamNames (list) -- Filters the results to only logs from the log streams in this list.

If you specify a value for both logStreamNamePrefix and logStreamNames , the action returns an InvalidParameterException error.

logStreamNames are not required field but it will return the result of this value and it only accepts the list value.

For your weird behavior of logStreamNamePrefix, it would need the / at last but I am not sure.

logStreamNamePrefix (string) -- Filters the results to include only events from log streams that have names starting with this prefix.

If you specify a value for both logStreamNamePrefix and logStreamNames , but the value for logStreamNamePrefix does not match any log stream names specified in logStreamNames , the action returns an InvalidParameterException error.

score 0 · Answer 2 · answered Mar 25 '21 at 14:09

Have you checked the next token return? This is not boto document but I think the reason is the same. https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html

It is stated that

This operation can return empty results while there are more log events available through the token.

I think the result is only partial. You will need to call the API again with the received "next token".

Couldn't fetch CloudWatch logs without LogStreamName specified

2 Answers2