6

I created a database table in AWS TimeStream which has a memory retention period of 7 days:

Retention

However, when I try to upload data that is 3 days old, it will be rejected. It seems that only data newer than 24 hours is accepted:

2020-10-26T13:43:33.244+01:00: Rejected a record: The record timestamp is outside the time range [2020-10-25T12:38:33.242Z, 2020-10-26T13:18:33.242Z) of the memory store.

Is this desired, or is there a configuration option I am missing?

rabejens
  • 7,594
  • 11
  • 56
  • 104

3 Answers3

6

Here's the relevant documentation: https://docs.aws.amazon.com/timestream/latest/developerguide/storage.html

Timestream loads data into the memory store only if the timestamp is within the timespan of its retention period. So if the retention period is 1 day, the timestamp can't be more than 1 day ago.

The way I understand how changes to the memory store retention period works... it takes effect from the point of when the change is made, so if you increase the retention period from 1 day to 7 days, you can't immediately insert data with a timestamp of 7 days ago.

I'm not sure if you had increased the memory store retention period and then tried to insert old data, but it sounds like that may have been the case?

Jason Capriotti
  • 1,836
  • 2
  • 17
  • 33
  • I first tried to increase the retention, then created a new table with a retention period of 7 days from scratch with the same result. – rabejens Nov 02 '20 at 08:07
  • @rabejens I just tried this method and it worked for me. Did you manage to solve the issue? – santamanno Aug 26 '21 at 00:02
  • In November 2021, they also released the ability to do magnetic writes. If your magnetic retention time is longer-- which is typical-- you can now write back that far. Note, you'll need to enable this table-by-table and magnetic writes can take up-to 6 hours from write to read. See the last section of this part of the docs: https://docs.aws.amazon.com/timestream/latest/developerguide/writes.html#writes.writing-data-inserts-upserts – Nadir Sidi Aug 05 '22 at 17:08
3

I'm a bit late to the party, but I had the same issue with one of my tables on AWS TimeStream. Changing memory retention policy after creation of the table was useless, it always kept the initial retention policy of 3 months.

I had to delete and recreate the table with the new memory retention policy in order to be able to insert the data accordingly.

Dharman
  • 30,962
  • 25
  • 85
  • 135
santamanno
  • 626
  • 4
  • 12
  • this was the answer for us too, had to drop and recreate the table. updating the table settings was useless – spaceman Apr 12 '23 at 08:28
2

Timestream has an issue when using unix epoch timestamps instead of milliseconds. I got around this by multiplying by 1000. I read this on https://medium.com/perlego/amazon-timestream-101-3b097db680cf

Maarten
  • 21
  • 1