How to trigger garbage collector to reduce database size?

Question

We use Xodus for a remote probe project in order to store temporary data before sending them to the centralized database. Thus, we have several stores which can grow or decrease accordingly to the environment (traffic, network connection, etc...). Thanks to the garbage collector, we expected to see decrease in the database file size but for the moment, it has only increased.

We tried several garbage collector configurations to trigger it as frequently as possible. For example, we have :

    conf.setGcFileMinAge(1);
    conf.setGcFilesInterval(1);
    conf.setGcMinUtilization(1);

Without visible effects...

After the store has been emptied, we expected to see reducing or deletion of .xd files but the database keeps growing and growing.

EDIT : I try to see GC effects with a simpler code as below :

        Environment exodus = Environments.newInstance(dbPath);

        final Transaction xtxn = exodus.beginExclusiveTransaction();
        Store store = exodus.openStore("testStore", StoreConfig.WITHOUT_DUPLICATES, xtxn);
        xtxn.commit();

        Thread.sleep(10 * 1000); // Wait to do actions after first  background cleaning cycle

        // Fill store, then clear it
        exodus.executeInExclusiveTransaction(tx -> {
            for(int i = 1; i <= 1000000; i++) {
                store.putRight(tx, LongBinding.longToEntry(i), StringBinding.stringToEntry(dbPath));
            }
        });
        clearStore(exodus, store);

        exodus.gc();
        Thread.sleep(5 * 60 * 1000); // Wait to see GC doing the work

    boolean clearStore(final Environment exodus, final Store store) {
        Transaction tx = exodus.beginExclusiveTransaction();
        try(Cursor cursor = store.openCursor(tx)) {
            boolean success = true;
            while(cursor.getNext() && success) {
                success &= cursor.deleteCurrent();
            }
            if(success) {
                tx.commit();
                return true;
            } else {
                log.warn("failed to delete entry {}", cursor.getKey());
                tx.abort();
                return false;
            }


        } catch(Exception e) {
            tx.abort();
            return false;
        }
    }

If I remove the first "sleep", Garbage Collector is doing the work, the database file size is reduced as expected, everything is ok. But if I keep the first "sleep", Garbage Collector never seems to be called. It's like the first background cleaning cycle is ok, but not the following ones... I keep default configuration in this example.

BTW, `conf.setGcMinUtilization(1)` allows 99% of free space in the database. Seems that is not what you expected. The higher value of the option, the more compact database. Probably, the javadoc for the option is not enough clear, I will review it. — Vyacheslav Lukianov, Jun 19 '19 at 08:30
And indeed for conf.setGcMinUtilization(1), we will try it with 90 in order to have 10% of free space allowed. Thanks ! — ATU, Jun 19 '19 at 12:54

score 1 · Answer 1 · answered Jun 19 '19 at 16:24

There is the Environment.gc() method. The javadoc for the method is as follows:

Says environment to quicken background database garbage collector activity. Invocation of this method doesn't have immediate consequences like freeing disk space, deleting particular files, etc.

I wouldn't recommend modifying default GC settings. EnvironmentConfig.setGcMinUtilization() can be used to keep the database more compact than it would be by default, or to decrease GC load (e.g., in parallel with batch updates). Basically, higher required minimum utilization (less admissible free space) results in higher GC load.

GC cleans the database file by file, selecting files with least utilization first. When a file is cleaned it is not deleted immediately, two conditions should be satisfied:

A delay configured by EnvironmentConfig.getGcFilesDeletionDelay() should pass. It's 5 seconds by default.
Any transaction (even read-only), created before the moment when the file is cleaned, should be finished (committed or aborted).

I retry with a simpler code example. I saw GB effects but only the first cycle. I edit the first post with my example. — ATU, Jun 20 '19 at 14:49
Also please take a look at `EnvironmentConfig.getGcStartIn()` option. By default, GC starts any its activity 10 seconds after the Environment is created. — Vyacheslav Lukianov, Jun 24 '19 at 12:54

How to trigger garbage collector to reduce database size?

1 Answers1