MVStore Online Back Up

Question

The information in the MVStore docs on backing up a database is a little vague, and I'm not familiar with all the concepts and terminology, so I wanted to see if the approach I came up with makes sense.

I'm a Clojure programmer, so please forgive my Java here:

// db is an MVStore instance
FileStore fs = db.getFileStore();
FileOutputStream fos = java.io.FileOutputStream(pathToBackupFile);
FileChannel outChannel = fos.getChannel();
try {
  db.commit();
  db.setReuseSpace(false);
  ByteBuffer bb = fs.readFully(0, fs.size());
  outChannel.write(bb);
}
finally {
  outChannel.close();
  db.setReuseSpace(true);
}

Here's what it looks like in Clojure in case my Java is bad:

(defn backup-db
  [db path-to-backup-file]
  (let [fs (.getFileStore db)
        backup-file (java.io.FileOutputStream. path-to-backup-file)
        out-channel (.getChannel backup-file)]
    (try
      (.commit db)
      (.setReuseSpace db false)
      (let [file-contents (.readFully fs 0 (.size fs))]
        (.write out-channel file-contents))
      (finally
        (.close out-channel)
        (.setReuseSpace db true)))))

My approach seems to work, but I wanted to make sure I'm not missing anything or see if there's a better way. Thanks!

P.S. I used the H2 tag because MVStore doesn't exist and I don't have enough reputation to create it.

i have created the tag for you – simbo1905 Dec 02 '19 at 21:13 — simbo1905, Dec 02 '19 at 21:13

score 0 · Answer 1 · answered Dec 02 '19 at 22:08

The docs currently say:

The persisted data can be backed up at any time, even during write operations (online backup). To do that, automatic disk space reuse needs to be first disabled, so that new data is always appended at the end of the file. Then, the file can be copied. The file handle is available to the application. It is recommended to use the utility class FileChannelInputStream to do this.

The classes FileChannelInputStream and FileChannelOutputStream convert a java.nio.FileChannel into a standard InputStream and OutputStream. There is existing H2 code in BackupCommand.java that shows how to use them. We can improve upon it using Java 9 input.transferTo(output); to copy the data:

    public void backup(MVStore s, File backupFile) throws Exception {
        try {
            s.commit();
            s.setReuseSpace(false);
            try(RandomAccessFile outFile = new java.io.RandomAccessFile(backupFile, "rw");
                FileChannelOutputStream output = new FileChannelOutputStream(outFile.getChannel(), false)){
                    try(FileChannelInputStream input = new FileChannelInputStream(s.getFileStore().getFile(), false)){
                        input.transferTo(output);
                    }
            }
        } finally {
            s.setReuseSpace(true);
        }
    }

Note that when you create the FileChannelInputStream you have to pass false to tell it to not close the underlying file channel when the stream is closed. If you don't do that it will close the file that your FileStore is trying to use. That code uses try-with-resource syntax to make sure that the output file is properly closed.

In order to try this, I checked out the mvstore code then modified the TestMVStore to add a testBackup() method which is similar to the existing testSimple() code:

    private void testBackup() throws Exception {
        // write some records like testSimple
        String fileName = getBaseDir() + "/" + getTestName();
        FileUtils.delete(fileName);
        MVStore s = openStore(fileName);
        MVMap<Integer, String> m = s.openMap("data");
        for (int i = 0; i < 3; i++) {
            m.put(i, "hello " + i);
        }

        // create a backup
        String fileNameBackup = getBaseDir() + "/" + getTestName() + ".backup";
        FileUtils.delete(fileNameBackup);
        backup(s, new File(fileNameBackup));

        // this throws if you accidentally close the input channel you get from the store
        s.close();

        // open the backup and verify
        s = openStore(fileNameBackup);
        m = s.openMap("data");

        for (int i = 0; i < 3; i++) {
            assertEquals("hello " + i, m.get(i));
        }
        s.close();
    }

With your example, you are reading into a ByteBuffer which must fit into memory. Using the stream transferTo method uses an internal buffer that is currently (as at Java11) set to 8192 bytes.

MVStore Online Back Up

1 Answers1