2

Theres a few dozen sqlite files i need to process and they are taking a few minutes each. Is there a way to copy or load the db into memory? IO (specifically reads as no writes are being code) is the bottleneck taking up 70% of the time.

Is there a short/quick solution i can use? Note: It will finish in <40minutes so if it isnt a quick solution it may not be worth the bother.

1 Answers1

1

Ok, this is a "way out there" idea, and with 40 min overhead may not be worth a shot.

This doc may be of help.Basically, you copy an on-disk DB to a memory one, similar to a Python example here, then dump it back to disk. Instead of many small writes, you'll be doing a couple large ones.

Again, some testing will be needed to check your specific situation and whether it will help.

Community
  • 1
  • 1
dawebber
  • 3,503
  • 1
  • 16
  • 16
  • this looks cool, but i dont know the syntax to copy it all from file to memory. I dont know how to do a insert into ThatDBOrFile from thisDBOrFile. –  Apr 26 '11 at 03:51
  • have you looked at the Python example I suggested. Here's a [Java example](http://code.ca-net.org/SQLite#In-memory_databases). Actually, my method is not that outlandish, just looks weird. – dawebber Apr 26 '11 at 03:56
  • +1. I dont have any of the functions i see in the python example such as iterdump and executescript. The java looks more similar but not the same. Its close enough. –  Apr 26 '11 at 04:04