0

With AWS SDK for C++ ListObjectsRequest, is there a way to have the SDK return the newest objects in the bucket based on the value of Set Marker? In the below code, the problem is that the objects in the object_list vector are newest to oldest. So, setting "last_key" to the key of the "back()" element results in 0 results the next time through in the loop (because there is nothing older than "last_key.")

If I set "last_key" to the key value of "front()" you end up looping over the same bucket objects over again.

Is there a way to use "last_key" that tells the SDK to get objects NEWER than "last_key"?

Aws::String last_marker;
while(true) {
  Aws::S3::Model::ListObjectsRequest objects_request;
  objects_request.WithBucket(bucket);
  if (last_key.length() > 0) {
     objects_request.SetMarker(last_key);
  }
  Aws::S3::Model::ListObjectsOutcome outcome;
  outcome = s_client.ListObjects(objects_request);
  Aws::Vector<Aws::S3::Model::Object> object_list = 
  outcome.GetResult().GetContents();
  //loop through object vector.
  last_marker.assign(object_list.back().GetKey());
}
user3472
  • 173
  • 9

1 Answers1

1

Typically, the marker you supply to a ListObjects call is the 'next marker' that was returned to you in the result of your previous call to ListObjects. It's a pagination marker. No, it doesn't support the notion of "later than", which is a timestamp-based concept, whereas the marker relates to S3 object keys, not timestamps.

Also note: you should use ListObjectsV2 rather than ListObjects.

jarmod
  • 71,565
  • 16
  • 115
  • 122
  • Thanks for the explanation. So then am I to ascertain that the only way to return objects newer in the bucket than the last time checked is to delete old ones? – user3472 Jul 20 '21 at 17:34
  • You can see the available parameters to ListObjects in the SDK documentation. There are no options that are timestamp-relative. You could potentially include timestamps somewhere in the S3 object key e.g. a prefix of 2021/07/20/ that would allow you to list objects for a specific date (and time if you included a time component). Also see this [client-side filtering solution](https://stackoverflow.com/questions/45429556/how-list-amazon-s3-bucket-contents-by-modified-date). – jarmod Jul 20 '21 at 19:15
  • I wonder why you can't add something like this "--query 'Contents[?LastModified > `'"$SINCE"'`]'" to the ListObjectsRequest. – user3472 Jul 30 '21 at 13:59
  • @user3472 You're referring to the awscli `--query` feature. This is a *client-side* feature. The ListObjects APIs do not implement this feature. – jarmod Jul 30 '21 at 14:09