Creating RocksDB SST file in Java for bulk loading

Question

I am new to RocksDB abd trying to create a SST file in Java for bulk loading. Eventual usecase is to create this in Apache Spark.

I am using rocksdbjni 6.3.6 in Ubuntu 18.04.03

I am keep getting this error,

org.rocksdb.RocksDBException: Keys must be added in order
    at org.rocksdb.SstFileWriter.put(Native Method)
    at org.rocksdb.SstFileWriter.put(SstFileWriter.java:104)
    at CreateSSTFile.main(CreateSSTFile.java:34)

The sample code is


 public static void main(String[] args) throws RocksDBException {
        RocksDB.loadLibrary();

        final Random random = new Random();


        final EnvOptions envOptions = new EnvOptions();
        final StringAppendOperator stringAppendOperator = new StringAppendOperator();
        Options options1 = new Options();
        SstFileWriter fw = null;
        ComparatorOptions comparatorOptions = new ComparatorOptions();

        try {

            options1 = options1
              .setCreateIfMissing(true)
              .setEnv(Env.getDefault())
              .setComparator(new BytewiseComparator(comparatorOptions));

            fw = new SstFileWriter(envOptions, options1);

            fw.open("/tmp/db/sst_upload_01");
            for (int index = 0; index < 1000; index++) {
                Slice keySlice = new Slice(("Key" + "_" + index).getBytes());
                Slice valueSlice = new Slice(("Value_" + index + "_" + random.nextLong()).getBytes());
                fw.put(keySlice, valueSlice);
            }

            fw.finish();
        } catch (RocksDBException ex) {
            ex.printStackTrace();
        } finally {
            stringAppendOperator.close();
            envOptions.close();
            options1.close();
            if (fw != null) {
                fw.close();
            }
        }
    }

If the loop index is less than 10 the file is created successfully and I was able to ingest that into rocks db.

Thanks in advance.

score 2 · Answer 1 · answered Nov 10 '19 at 09:58

I think I found the problem with the code.

The keys must be in order for the SST. The way I do the looping and using String lexicographical comparison for ordering, produces incorrect ordering. Like comparing "10" and "9" would break the order. Instead of that, if I sort all the keys before inserting into SST file it works.

Map<String, String> data = new HashMap<>();
            for (int index = 0; index < 1000; index++) {
                data.put("Key-" + random.nextLong(), "Value-" + random.nextDouble());
            }

            List<String> keys = new ArrayList<String>(data.keySet());

            Collections.sort(keys);

            for (String key : keys) {
                Slice keySlice = new Slice(key);
                Slice valueSlice = new Slice(data.get(key));
                fw.put(keySlice, valueSlice);
            }

When I tried with integer keys I found the issue.

score 1 · Answer 2 · answered Feb 16 '22 at 10:04

1

The keys in sstfile should be in increasing order. so you can start from index=10 it will work.

answered Feb 16 '22 at 10:04

LOGESH RAMESH

11
1

score 0 · Answer 3 · answered Nov 18 '20 at 02:25

In fact ,you can create Comparator extends DirectComparator to avoid sorted.

class MyComparator extends DirectComparator {

        public MyComparator(ComparatorOptions copt) {
            super(copt);
        }

        @Override
        public String name() {
            return "MyComparator";
        }

        @Override
        public int compare(DirectSlice a, DirectSlice b) {
            // always true
            return 1;
        }
    }
}

then set option

options1.setComparator(new MyComparator(comparatorOptions));

Creating RocksDB SST file in Java for bulk loading

3 Answers3