1

I'd like to use java to work with a data structure that's going to end up over 100 GB. I need to write and read from the thing in 1:1 ratio, as many times per second as possible. In Java form it's a HashMap>. Can I keep and operate on it in disk as if it were in RAM? Can this double as a persistence scheme?

trincot
  • 317,000
  • 35
  • 244
  • 286
brandones
  • 1,847
  • 2
  • 18
  • 36
  • What operating system are you using? (Be as specific as you possibly can. Include whether it's 32-bit or 64-bit.) But my answer will be this: Write a data structure that does exactly what you want. Don't try to shoehorn an existing structure into doing something it wasn't designed to do. – David Schwartz Aug 16 '12 at 01:13
  • Take a look at [this question](http://stackoverflow.com/q/1023200/335858). – Sergey Kalinichenko Aug 16 '12 at 01:15
  • You could take a look at http://stackoverflow.com/questions/3316630/java-disc-based-hashmap – MadProgrammer Aug 16 '12 at 01:22
  • Perhaps, some sort of a database? Or a one of the fast NoSQL solutions? – Strelok Aug 16 '12 at 01:38
  • Checkout LevelDB by Google (http://code.google.com/p/leveldb/). You will need this library (https://github.com/fusesource/leveldbjni) to call it from Java. – Strelok Aug 16 '12 at 01:45

3 Answers3

2

There are "cache" packages that can help you out here. Ehcache comes to mind. It will allow you to create a cache which stores a finite # of objects in memory, and overflows additional objects to disk. This is all done behind the scenes, so all you have to do is get/put from the cache.

Matt
  • 11,523
  • 2
  • 23
  • 33
0

You could, but you'd probably want to first understand the access patterns. If you're seeking randomly a whole lot and you're not using an SSD, you're going to incur milliseconds of wait for every IO. If you're only seeking within certain regions, you may want to just slam those parts into memory. Try to organize its data to be as clustered on your access pattern as possible.

Kevin Hsu
  • 1,726
  • 11
  • 14
0

No you cannot use a disk file as heap in Java. You can map a file into the address space, but it won't be part of the heap; i.e. you won't be able to use it to hold regular Java objects.

The simplest way to do this (if you have enough RAM and you are running a 64bit OS and 64bit JVM) is to simply give the JVM a huge heap. But I guess this is not an option for you. So you options are going to be:

  • use a conventional database
  • use an existing off-the-shelf data caching products for Java
  • attempt to roll your own data caching.
Stephen C
  • 698,415
  • 94
  • 811
  • 1,216