3

Thanks in advance!

I am planning to use AWS RDS Postgres for pretty big data (> ~50TB) , but I have couple of questions un-answered

  1. Is 16TB the maximum limit for AWS RDS Postgres instance, if so how do people store > 16TB data.

  2. Is the limit of 16TB for RDS the maximum database size post compression that Postgres can store on AWS.

  3. Also I do not see any option for enabling compression while setting up AWS RDS Postgres DB instance. How to enable compression in AWS RDS Postgres?

I have followed :

  1. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html

  2. https://blog.2ndquadrant.com/postgresql-maximum-table-size/ (wherein Postgres table can have size greater than 32TB).

  3. https://wiki.postgresql.org/wiki/FAQ#What_is_the_maximum_size_for_a_row.2C_a_table.2C_and_a_database.3F

user1393608
  • 1,299
  • 3
  • 16
  • 29
  • 1
    That is an arbitrary limit imposed by Amazon. Maybe it would be a good idea to have your database elsewhere or even manage it yourself. Large databases like that are hard to handle (just think of backup) - that is so in general, it is not a PostgreSQL limitation. – Laurenz Albe Feb 11 '19 at 08:01
  • Thanks Laurenz, but how can one scale if one has more than 50TB data size and wants to use AWS RDS. – user1393608 Feb 11 '19 at 09:44
  • Split the data in several databases? Maybe ask Amazon what they offer. Why are you fixed on that one provider? – Laurenz Albe Feb 11 '19 at 09:49
  • Is this a problem that you actually have now, or a problem that you hope to have someday? – kdgregory Feb 11 '19 at 21:59
  • I might have data in excess of the limit that AWS RDS imposes, also can you please throw some light on point number 3 in my question (compression) – user1393608 Feb 12 '19 at 03:45

3 Answers3

5

In addition to RDS for PostgreSQL, which has a 32 TiB limit, you should take a look at Amazon Aurora PostgreSQL, which has a 64 TiB limit. In both cases, the largest single table you can create is 32 TiB, though you can't quite reach that size in RDS for PostgreSQL as some of the space will be taken up by the system catalog.

Full disclosure: I am the product manager for Aurora PostgreSQL at AWS.

Kevin J
  • 91
  • 1
  • Then why does this say 32 TiB? https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Limits.html#RDS_Limits.FileSize.Aurora – Kongress Aug 05 '20 at 15:20
3

As of 2019/02/11, Amazon's docs state that the maximum database size for a Postgres RDS instance is 32TiB. Additionally, this 32TiB limit appears to be a hard limit (Some AWS limits are considered 'soft' in that a user can request that they be raised)

As others have suggested, one option would be to manage your own database, however this is not likely to be easy given the scale of data we are talking about here.

Staying within the AWS ecosystem, another option would be to store all your data in S3, and use AWS Athena to run queries, although depending on what sort of queries you'll be running, it could get quite expensive.

As for your question about compression, if you are storing data in s3, you can compress it before you upload. You might also find this answer helpful.

Matt
  • 968
  • 2
  • 13
  • 22
2

RDS limit has been revised to 64 TB.

MariaDB, MySQL, Oracle, and PostgreSQL database instances: 20 GiB–64 TiB

SQL Server for Enterprise, Standard, Web, and Express editions: 20 GiB–16 TiB

sumitskumar
  • 21
  • 2