3

I was looking at AWS QLDB service to store the audit trail history of changes that were made to our application, so that it can be immutable.

But, at the end it is a database & we can't just keep on adding data (storing such large amount of data is costly).

At some point in time we will need to roll over / archive existing data & start over everything a fresh.

Was wondering how AWS QLDB will be able to handle such scenarios ?

P.S. I am a newbie to AWS QLDB.

NoSQLKnowHow
  • 4,449
  • 23
  • 35
Srujal Kachhela
  • 209
  • 1
  • 4
  • 15

2 Answers2

3

as every business have limitations on amount of history that can be stored

I would argue that this assertion too broad.

There are business cases that need to be able to document historical events indefinitely (or indefinitely, for all practical purposes) because either the data remains relevant or because without retaining the entire history, there is no way to conclusively prove that the current state of the database is as it should be... and that is the purpose of QLDB -- maintaining historical records that cannot be modified or deleted, either accidentally or on purpose.

With QLDB, your data’s change history is immutable – it cannot be altered or deleted – and using cryptography, you can easily verify that there have been no unintended modifications to your application’s data.

https://aws.amazon.com/qldb/

Each transaction builds on the one before it. Oversimplified, it lools like this:

hash(t1) = SHA256(t1)
hash(t2) = SHA256(t2 + hash(t1))
hash(t3) = SHA256(t3 + hash(t2))
...

Those hash values are also stored, so each transaction can be cryptographically verified against its predecessor, all the way back to the beginning of time. Deleting older records removes information necessary for verifying newer records.

A use case where you plan to purge historical data seems like an incorrect application of QLDB.

Michael - sqlbot
  • 169,571
  • 25
  • 353
  • 427
  • From your formula, you could store the digest in a particular point in time and delete the data up to that point and still be able to verify future data. Can QLDB handle cases like this? – Alko Sep 30 '19 at 07:00
  • @Alko, storing an intermediate hash as a starting point still defeats the purpose, because you would then have no possible way to justify having stored that hash as the starting value -- the purged data would be necessary to establish the validity of that hash. The idea here is that the entire collection of data -- from the beginning, with zero records up to the present state -- can be proven to be immutable. This is the quintessence of QLDB. – Michael - sqlbot Sep 30 '19 at 11:49
  • depens on the usecase. For example in an supply chain application, there is no need to hold data for a yogurt that expired years ago. You only need store the yogurt data a month after it expired and after that its not needed anymore and would only make data storage more expensive over time. A proof of existance (the yogurts hash) is sufficient in this case. – Alko Oct 07 '19 at 09:14
0

In AWS Docs it is mentioned that:

Amazon QLDB is a fully managed ledger database that provides a transparent, immutable, and cryptographically verifiable transaction log ‎owned by a central trusted authority. Amazon QLDB tracks each and every application data change and maintains a complete and verifiable history of changes over time.QLDB is easy to use because it provides developers with a familiar SQL-like API, a flexible document data model, and full support for transactions. QLDB is also serverless, so it automatically scales to support the demands of your application. There are no servers to manage and no read or write limits to configure. With QLDB, you only pay for what you use.

For more further reference, you can refer this link: https://aws.amazon.com/qldb/
Thanks

Aress Support
  • 1,315
  • 5
  • 12
  • 2
    This doesn't seem to address the question. "You only pay for what you use" includes *storage*, which is what OP wishes to avoid paying for, indefinitely. – Michael - sqlbot Mar 27 '19 at 18:03