13

I recently started to work with AWS Data Migration Service (DMS) and running into some issues.

Currently attempting to migrate a 10GB Oracle DB to AWS RDS Postgres. Works but has crazy(?) memory requirements. Feels like it loads the entire DB into memory... Started with dms.r4.large (15.5GB) but can not allocate memory after approx. 98%.... Will run smoothly with dms.r4.xlarge (30.5GB)

memory consumption during replication task run

As you can see in the screenshot (free-able memory, minimum), the instance is constantly running "full" before all memory gets released when the task finishes (or crashs).

Is there any setting to change this and why does it behave like this? It makes the whole task unnecessary expensive...

gapvision
  • 999
  • 1
  • 10
  • 30

1 Answers1

17

As confirmed by AWS, this was indeed a bug with the latest engine (v3.1.3). Following additional insights have been provided by AWS to estimate the actual memory requirements:

Full LOB mode (using single row insert+update, commit rate)

Memory: (# of lob columns in a table) x (Number of table in parallel, default is 8) x (lob chunk size) x (Commit rate during full load) = 2 * 8 *64(k) * 10000k

Note: You may consider to reduce the "Commit rate during full load " value because we allocate memory using roughly the above method

Limited LOB mode (using array)

Memory: (# of lob columns in a table) x (Number of table in parallel, default is 8) x maxlobSize x bulkArraySize = 2 * 8 * 4096(k) * 1000

gapvision
  • 999
  • 1
  • 10
  • 30
  • 1
    what is bulkArraySize? – dev_etter Oct 02 '19 at 03:18
  • 2
    I would assume it equals to commit rate (bulk size) – gapvision Oct 03 '19 at 07:07
  • I thought this was the case as well. Thank you for sharing this formula, it was a lifesaver to me this week. – dev_etter Oct 03 '19 at 17:45
  • Just found out that InlineLobMaxSize is interchangeable with LobChunkSize wrt to this formula. So if you thought to be clever and set InlineLobMaxSize to the max, now you know why your task is running out of memory. Also, can confirm @gapvision that bulkArraySize == commitRate for this formula. – John Jones Jan 31 '23 at 23:40