How to avoid getting older results in AWS cloud trail API call

Question

We are using AWS Cloud Trail to retrieve data (cloud trail events). We have used the gem 'aws-sdk-cloudtrail' (version 1.0). As per Cloud Trail we are able to retrieve maximum 50 results (latest once). To fetch the previous (older once ) results we use 'next-token' received in the previous response. We perform this until we get an empty 'next-token'. When we receive an empty token it means all the cloud trail data has been retrieved.

For example: Suppose Cloud Trail has 100 events logged in: In the first api call we received the latest 50 results along with the token to retrieve next 50 (older 50). In the second api call we receive the remaining 50 results (older results) along with next token as nil. It means there are no more results to be fetched.

In our case we are saving all the results received from the trail in our local database.We repeat this periodically. When doing it for the second time (repeating above explained process) we again receive few newer and few older results. We again repeat the API call until we get 'next-token' as nil. This results in receiving redundant data which was already stored in the database when the first cycle executed. Is there any way to get only the newly logged cloud trail events second cycle on-wards.

viks · Answer 1 · 2018-04-12T05:41:55.273

Like @Vorsprung said you can use the max event date time from your local database.

Here is the detailed solution for your Use case/Problem:

1. Query to your local database to check that cloudtrail data is present in the local database.

    IF yes 
        // It means you have stored some data from cloudtrail before.
        // And now you are going to do request to cloudtrail for new trail events.
        // Note - At a time of the first request you don't have a token (i.e. next-token)

        GOTO Step 3

    ELSE
        // It means you have not stored any data from cloudtrail before.
        // And now you are going to do the first request to cloudtrail. 
        // Note - At a time of the first request you don't have a token (i.e. next-token)

        GOTO Step 2

2.  LOOP true

        token = nil

        IF token
            // Send request to cloudtrail to get next bactch of latest cloudtrail events, now pass token(i.e. next-token) as parameter.
            // Which will return the maximum latest 50 trail events.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END

        ELSE
            // Send request to cloudtrail to get the latest cloudtrail events.
            // Which will return the maximum latest 50 trail events.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END
        END
    END

3.  LOOP true

        token = nil
        start_date_time = max_trail_event_date_time_form_local_db

        IF token
            // Send request to cloudtrail to get next bactch of latest cloudtrail events, now pass token and start_date_time(i.e. next-token and max_event_date_time_form_local_db) as parameters.
            // Which will return the maximum latest 50 events which are logged after start_date_time.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END

        ELSE
            // Send request to cloudtrail to get the latest cloudtrail events, now pass start_date_time(i.e. max_trail_event_date_time_form_local_db) as parameter.
            // Which will return the maximum latest 50 events which are logged after start_date_time.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END
        END
    END

Hope it will be helpful.

score 0 · Answer 2 · answered Mar 13 '18 at 15:05

0

Select max date from your local database and then use that as the start date for the cloudtrail events

answered Mar 13 '18 at 15:05

Vorsprung

32,923
5
39
63

score 0 · Answer 3 · answered Mar 13 '18 at 17:54

You save the "NextToken" in your local database and pass it when you call the API next time. Here is an example.

import boto3

cloudtrail = boto3.client('cloudtrail')
paginator = cloudtrail.get_paginator('lookup_events')

StartingToken = None

page_iterator = paginator.paginate(
    LookupAttributes=[{'AttributeKey':'EventName','AttributeValue': 'RunInstances'}],
    PaginationConfig={'PageSize':10, 'StartingToken':StartingToken })
for page in page_iterator:
    for event in page["Events"]:
        print(event["EventName"],event["EventTime"])
    try:
        token_file = open("token","w") 
        token_file.write(page["NextToken"]) 
        StartingToken = page["NextToken"]
    except KeyError:
        exit()

How to avoid getting older results in AWS cloud trail API call

3 Answers3