43

I have a CloudWatch Logs Insights query, which shows "7000 records matched", but when I try Actions -> Download query results (CSV), only 1000 records are exported (same as shown in the console). I cannot find any way to export the "full" query results. Am I missing anything?

lznt
  • 2,330
  • 2
  • 22
  • 27

4 Answers4

51

Adding

| limit 7000

to the end of the query fixed the issue

The max is 10000 according to https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_StartQuery.html#API_StartQuery_RequestSyntax, so if my query has more than 10000 records, there seems to be no way to get the complete data.

lznt
  • 2,330
  • 2
  • 22
  • 27
  • 1
    If you need more than 10000 results and you happen to code in Go, take a look at the Incite library on GitHub: https://github.com/gogama/incite, comprehensive GoDocs [here](https://pkg.go.dev/github.com/gogama/incite). It has a lot of nice features, and one includes automatic support for arbitrarily chunking the results. You just tell it the whole time range you need and the chunk size, and it will issue as many queries as it needs (in the background, hidden, in parallel where possible) and give you a unified stream containing all the merged results. – 0xbe5077ed Sep 14 '21 at 15:07
  • @0xbe5077ed how about aggregated data over some time period (i.e. stats count(*) by field), chunking the result by time ranges won't work in this case – Dima Svider Sep 15 '21 at 22:39
  • Great question @DimaSvider. There's no easy way to work around this other than by doing client-side aggregation. The [Incite](https://github.com/gogama/incite) library can still help with this, for example it makes it much easier to convert your query output into a usable format that you can use for client-side aggregation. The same is true of sorted queries (using Insights `| sort ...` command), there's no way around some post-processing. (**Comment 1/2**, to be continued...) – 0xbe5077ed Sep 16 '21 at 17:11
  • 1
    (**Comment 2/2**, continued.) @DimaSvider, the other thing to keep in mind is that too my knowledge, the limitation is 10K **results**. Specifically for aggregated queries since the results are already aggregated in the service, it tends to be a lot harder to hit the 10K limit. – 0xbe5077ed Sep 16 '21 at 17:13
6

This is an addition to lznt's answer, addressing the following issue:

if my query has more than 10000 records, there seems to be no way to get the complete data.

I think there is a workaround: You can mess with the time range of the query. Order the results by timestamp. You then know the timestamp of the last record that you still managed to get.

For the next query, set the time range of the query so that you start where the previous query ended.

It is admittedly an ugly, iterative workaround, with a human in the loop.

Disclaimer: I haven't tested it.

Ali
  • 56,466
  • 29
  • 168
  • 265
  • For low volume this may work, but it's tedious. Falls apart when you pass 10000/sec since the UI only allows increments of whole seconds. – Gus Oct 28 '22 at 20:52
  • @Gus *"when you pass 10000/sec"* Hahaha, I would love to have that problem! Joke aside: As I wrote in the answer, it is an ugly workaround. – Ali Oct 28 '22 at 21:09
  • @Gus By the way, Cloudwatch Log Insights was a big disappointment for me, and in the end, I did not use it at all. It was flaky, slow, and unreliable. I simply download all the data and do the processing myself locally, on my machine. – Ali Oct 28 '22 at 21:14
3

True that max is 10k

But

datetime

can be narrowed, as well as log group for AWS CloudWatch Logs Insights

CodeFarmer
  • 2,644
  • 1
  • 23
  • 32
  • If you need more than 10000 records and can write code in Go, take a look at the Incite library on GitHub: https://github.com/gogama/incite, comprehensive GoDocs [here](https://pkg.go.dev/github.com/gogama/incite). It has a lot of nice features, and one includes automatic support for arbitrarily chunking the results. You just tell it the whole time range you need and the chunk size, and it will issue as many queries as it needs (in the background, hidden, in parallel where possible) and give you a unified stream containing all the merged results. – 0xbe5077ed Sep 14 '21 at 15:08
1

As stated in some of the other answers, you can only export up to 10,000 lines from CloudWatch Log Insights. However, you can export the entirety of the log group's contents to S3 and then subsequently filter for the log files that you want with a little bit of munging. This post shows how to do that, and includes a Gist for consolidating the exported results.

phenri
  • 71
  • 1
  • 5