1

Now I'm using berkeley DB to build a Btree cotaining 80 million records. I need to insert them in, scan them, and update their values. However, only the update takes around 1 hours, which is too slow for me. In https://docs.oracle.com/cd/E17076_05/html/gsg/CXX/BerkeleyDB-Core-Cxx-GSG.pdf, it is said each read operation in Berkeley DB would load a whole page into memory. I wonder if there are any methods could avoid it? That is, every time I want to get a record, only its pair of key and data will be loaded into memory rather than the whole page containing it? Thank you.

Kaijie Zhu
  • 51
  • 3
  • 1
    Hi Kaijie, welcome to Stack Overflow! It sounds like you may be operating near the edge of what BDB can do, but it totally depends on the size of your records. At 80 million records per hour, you're at about 45 microseconds per record. This might be "pretty good" if your records are kilobytes in size. What's the size range for your keys and your data? And, what do you have the BDB page size set to? You absolutely have to read in the entire leaf page, and possibly several index pages. But you can change the page size to better fit your records. – Mike Andrews Sep 30 '18 at 02:08
  • @MikeAndrews Thanks Mike. My key is a struct composed of 3 unsigned int (I think it is 12 Bytes). and for value, it includes an unsigned int and an void pointer(8 Bytes I think, the pointer is always set to NULL for later extend). BDB page is set to 4KB, the block IO size of my server. Do you have any opinions for me to improve the efficiency? Thank you. – Kaijie Zhu Oct 01 '18 at 09:03
  • @MikeAndrews By the way, the "index preparation" actually incluses 1 insertion and 2 updates. That is, it constains of 3 steps: (1) insert all the 80 million records into the dataset (2) scan from the beginning of the record in index, and update their value one by one (3) another scan from the beginning and update theira value one by one. The memory is limitted to only 2GB so I think IO operation now is the botthle neck of the program. And the size of the dataset is actually 2GB.After all inserted ,the .db file is around 5GB. – Kaijie Zhu Oct 01 '18 at 09:17
  • You'd have to make the call for your application, but one way to improve write performance is to tune down how aggressively atomic the database is. If you don't need atomicity at all, you could run without transactions. If you need atomicity, but can tolerate a few lost records on recovery, try the `DB_TXN_WRITE_NOSYNC` flag, or even `DB_TXN_NOSYNC` (see this answer: https://stackoverflow.com/questions/3825022/optimizing-put-performance-in-berkeley-db). – Mike Andrews Oct 01 '18 at 13:28
  • And, it's great that the page size matches the block I/O size for your server! I wouldn't change that. If your records were large, and you were constantly overflowing, you could go to a bigger page size. But, 4K is about as small as you'd want to go, even for very small key/data, like you have. – Mike Andrews Oct 01 '18 at 13:31
  • Another option, though this starts defeating quite a bit of the point of having a database like this, is to "bin" your records into larger logical records. If you could group your records into bins of 64, sort of treating BDB like a hash table, then you'd cut the number of inserts down from 80 million to 1.25 million. For your application, that record would be around a kilobyte, and still easily fit on a page. – Mike Andrews Oct 01 '18 at 13:36
  • @MikeAndrews Thanks. I didn't use DBenv. So I think there should be no transactions. Does it? – Kaijie Zhu Oct 01 '18 at 17:48
  • Ah, well, there are certainly no logs without an environment. You'd think that'd be the best case for Berkeley DB! (as an aside, make sure you've set your cache size with DB->set_cachesize().) But, if you're not using transactions or an environment... maybe you don't want Berkeley DB? I mean, you *can* make BDB run fast, but you'll be spending a bunch of your life energy on it. You might evaluate LMDB, or another modern key/value store instead? – Mike Andrews Oct 01 '18 at 18:31
  • @MikeAndrews En, I‘m sorry. I didn't use the DB->set_cache and the default size of that for BDB is 256KB. Does this matters? I don't completely understand its impact. – Kaijie Zhu Oct 02 '18 at 20:05
  • Oh, yeah! You’ll want to set that as large as you can tolerate. If you’ve got 2GB of RAM, allow space for the os and for whatever else is on the system. If you can get it above a 1GiB, your program should be much, much happier. – Mike Andrews Oct 03 '18 at 01:28

0 Answers0