5

As you all know, AWS Timestream was made generally available in the last week.

Since then, I have been trying to experiment with it and understanding how it models and stores the data.

I am facing an issue in ingesting records into Timestream. I have certain records dated 23rd April 2020. On trying to insert these records into a Timestream table, I get the RecordRejected error. According to this link, a record is rejected if it has the same dimension, timestamp or if the timestamp is beyond the retention period of the memory store of the table.

I have set the retention period of the memory store of my table to 12 months. According to the documentation: any records having a timestamp beyond 12 months would be rejected. However, the above mentioned record gets rejected despite having a timestamp within 12 months from now.

On investigating further, I have noticed that, records with today's date (5th Oct 2020) get ingested successfully, however, records with a date 30 days before do not get ingested, i.e. 5th September 2020. To ensure this, I have also tried inserting a record with the date 6th Sept and a few more days between today's date and 5th Sept. All these are getting inserted successfully.

Could somebody explain why I am not able to insert records having a timestamp within the retention period of the memory store? It only allows me to insert records that are at the most 30 days old. I would also like to know if there is a way we could insert historical data directly into the magnetic store. The memory store retention period may not be sufficient for my use case and I may need to insert data that is 2 years old or more. I understand this is not a classic use case of timestream, but I am still curious to know.

I am stuck on this issue and would really appreciate some help.

Thank you in advance.

Carl Manaster
  • 39,912
  • 17
  • 102
  • 155
ShwetaJ
  • 462
  • 1
  • 8
  • 32

2 Answers2

1

I had a very similar issue, and for me it turned out that I had to set the Memory Store Retention Period to 8766 hours - which is slightly MORE than one year. I've no clue why that is, and why it works, but it worked for me importing older data.

PS: I'm pretty sure it's a bug in timestream PPS: I've found the value by using the "default" set in the aws console. No other value worked for me.

Blitz
  • 5,521
  • 3
  • 35
  • 53
  • Is there a way we can flag this error? I have posted on a forum too, but there is no dedicated channel for timestream on aws forum. I will try using the hours metric and use your value, will check if that works. Nonetheless, the behavior should be consistent, whether you specify it in months or hours. – ShwetaJ Oct 12 '20 at 08:01
  • By `TTL` do you mean memory store retention or magnetic store retention please? – variable Nov 04 '20 at 11:48
  • Memory store retention period. – ShwetaJ Nov 04 '20 at 12:07
  • Did any one know about using timestream in php ? https://stackoverflow.com/questions/65027337/how-to-setup-amazon-timestream-in-php – Ayush Srivastava Nov 30 '20 at 15:50
  • @AyushSrivastava Sorry, haven't tried in php. The AWS SDK for PHP should have something for this. – ShwetaJ Jan 28 '21 at 23:29
0

Timestream loads data into the memory store only if the timestamp is within the timespan of its retention period. So if the retention period is 1 day, the timestamp can't be more than 1 day ago.

AWS TimeStream: Records that are older than one day are rejected

Lior Mechlovich
  • 1
  • 3