3

We use ArangoDB 3.3.14 (Community Edition) with MMFiles storage engine for a relatively large data set (a bit over 30 GB when you back it up). We run it inside of a docker container using ECS. Our host VM has 64 GB of RAM and we have dedicated 55 GBs exclusively for ArangoDB container (we set a hard limit for that container to 55 GBs).

When ArangoDB is just started and have all the collections loaded into the RAM it would take about 45 GBs, so we have about 10 GBs of free RAM to be used for queries, etc.

The problem is that after some period of time (depending on usage) ArangoDB eats all the 55 GB of RAM and does not stop there. It continues to consume the RAM over the set hard limit and at some point, docker kills the container with exit code 137 and the status reason OutOfMemoryError: Container killed due to memory usage.

The restart causes a lot of problems for us because we need to wait until all the collections and graphs are loaded back into the RAM again. It takes about 1-1.5 hours for our data set and you can not use ArangoDB while it is "restarting".

My question is how can I limit ArangoDB RAM usage, let's say to 54 GBs, so it never reaches a hard memory limit set for a docker container?

Peter Liapin
  • 1,196
  • 9
  • 20
  • Did you tried `--javascript.gc-frequency frequency` ? This might reduce the charge if no answer is provided. Or read https://stackoverflow.com/questions/24380071/memory-usage-of-arangodb – Gilles-Antoine Nys Jan 29 '19 at 08:38
  • Yes, I have seen that topic, but it does not answer my question which is how do we limit overall ArangoDB RAM usage so it never use more RAM than we allow it to use. I have not tried changing gc-frequency, but I guess it would not solve the problem too as it does not actually limit a RAM consumption. Yes, it may probably help ArangoDB to survive longer, but would not solve the real problem. Anyway, thank you for a proposal, we will try it. – Peter Liapin Jan 29 '19 at 08:46

2 Answers2

2

In 3.3.20, ArangoDB introduces the parameter {{total-write-buffer-size}} which limits the write buffer. You can try adding this to your configuration file:

[rocksdb]
block-cache-size = <value in bytes>  # 30% RAM
total-write-buffer-size = <value in bytes>  # 30% RAM
enforce-block-cache-size-limit = true

[cache]
size = <value in bytes>  # 20% RAM

or you can pass parameter to the command line:

arangod --cache.size <value in bytes>  # 20% RAM \
    --rocksdb.block-cache-size <value in bytes>  # 30% RAM \
    --rocksdb.total-write-buffer-size <value in bytes>  # 30% RAM \
    --rocksdb.enforce-block-cache-size-limit true

You can also tune how much memory assign per single component as per your usage. But you have to upgrade at least to 3.3.20.

Luigi Servini
  • 36
  • 1
  • Hello, Luigi, sorry, I had to clarify that we still use MMFiles files storage engine (not RocksDB). Do you have a similar set of settings for MMFiles engine? I know RocksDB is a new default way and is recommended, etc. but we tested our relatively large data set with a new RocksDB and found that the performance for most of our AQL queries (traversals) decrised dramatically. Some of the queries which used to take about 10 seconds with MMFiles now take forever with new RocksDB engine and I have to kill them after some period of time. So, RocksDB does not look as a good option for us just yet. – Peter Liapin Jan 29 '19 at 17:57
  • Hello Peter, yes, right, those specific parameters are for RocksDB (apart --cache.size). Probably in your case is better to move to ROcksDB, which has several advantages: * document-level locks * support for large data-sets * persistent indexes And you can limit the memory consumption as well (starting from 3.3.20 on Linux). With MMFILES both collections and indexes have to fit in memory. – Luigi Servini Jan 29 '19 at 18:05
  • Yes, you cannot limit because all data structures have to fit in memory (collections + indexes). – Luigi Servini Jan 29 '19 at 18:21
  • Both collections and indexes do fit into the RAM nicely. Please, see my comments to you other answer above. Let's continue our conversation there. – Peter Liapin Jan 29 '19 at 18:26
0

yes, right, those specific parameters are for RocksDB (apart --cache.size). Probably in your case is better to move to ROcksDB, which has several advantages:

  • document-level locks
  • support for large data-sets
  • persistent indexes

And you can limit the memory consumption as well (starting from 3.3.20 on Linux). With MMFILES both collections and indexes have to fit in memory.

Luigi Servini
  • 36
  • 1
  • Sorry, this does not answer my question. Please note that all our collections fit into the RAM nicely. Our container have 55 GBs of RAM while all the collections and graphs take just about 45 GBs. My problem is not related to the collections not fitting into the RAM, it is about extra RAM which ArangoDB takes for its internal needs (when we run AQL queries, update, etc). My question is how can I limit that extra memory consumed by ArangoDB for some other needs (not for the real data)? – Peter Liapin Jan 29 '19 at 18:23
  • Also, please note that RocksDB does not look like a good option for us yet because the performance of our AQL queries deceases dramatically with RocksDB. I will add a separate discussion here on StackOverflow about that problem. So, does it mean that with the MMFiles engine we have no options to limit RAM usage? – Peter Liapin Jan 29 '19 at 18:24
  • Hi Peter, is 45GB the dump size? In that case, you have to consider also indexes, which are built in memory at startup. You can also try to limit --cache.size (by default is around 25% of RAM). This cache is used for some point lookup operations, e.g. when fetching edges by their _id value(s). – Luigi Servini Jan 30 '19 at 09:34
  • Hi Luigi, no 45GB is the amount of RAM consumed by ArangoDB when everything is loaded, i.e. all the indexes are already taken into account. In my original question above I mentioned that the backup of that database is a bit over 30GB. I do understand that the size of the database is not the same as the amount of RAM it actually needs because of the indexes. – Peter Liapin Jan 30 '19 at 14:45