3

I'm testing data structure performance with very large data.

As a temporary workaround (see here) I want to write memory to disk.

I want to test with very big datasets - how can I make it so that when the java VM runs out of memory it writes some of it to disk?

Community
  • 1
  • 1
Justin
  • 447
  • 4
  • 10
  • 33
  • Sorry, I'm confused,... are you using a database? – Hovercraft Full Of Eels Apr 04 '13 at 00:42
  • No, I'm testing van emde boas trees and other data structures, and everything is in memory. Is this still possible? – Justin Apr 04 '13 at 00:45
  • 1
    I don't think you can tell the JVM to use a file when RAM is full. You will have to write your own data structure which reads from/writes to a file, possibly buffering data. – Code-Apprentice Apr 04 '13 at 00:46
  • You can use [WeakHashMaps](http://docs.oracle.com/javase/6/docs/api/java/util/WeakHashMap.html) but you will have to make sure you only use WeakReferencese in other locations as well. – BevynQ Apr 04 '13 at 00:48
  • 2
    Are you stuck using 32-bit Java? I'm not a Java dev, but I expect that if you were running a 64-bit process, you would be able to address a lot more memory, plus your OS' paging system would also handle disk swapping for you... – Simon MᶜKenzie Apr 04 '13 at 00:54
  • I'm using 64-bit java (or so it says when I make the call to check). – Justin Apr 04 '13 at 01:19
  • ... Then it sounds like you need a bigger paging file. 64-bit should allow you to address something in the order of [256TB](http://stackoverflow.com/questions/2093679/max-memory-for-64bit-java)... – Simon MᶜKenzie Apr 04 '13 at 03:19
  • 1
    Possible duplicate of http://stackoverflow.com/questions/4815633/are-they-any-decent-on-disk-implementations-of-javas-map# – Jim Garrison Apr 04 '13 at 05:45

2 Answers2

0

Since we're talking about temporary fixes here you could always increase your page file if you need a little extra space (swap file in most linux distros)

Here's a link from Microsoft: http://windows.microsoft.com/en-us/windows-vista/change-the-size-of-virtual-memory

Linux: http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

Now let me say that this isn't a good long term fix, but I understand that sometimes developers just need to make it work. If this is something that will ever see a production environment you may want to look at a tool like Hadoop. It allows you to distribute your data processing over multiple JVM's--a tool built for a "big data" application like the one you're describing

darkpbj
  • 2,892
  • 4
  • 22
  • 32
0

Maybe you can use stream, or some buffered one. I think that will be the best choice for testing such structure. If you can read from disk using stream and that will be not make any additional objects(only that which are necessary) so you can have all jvm memory for your structure. But maybe you can describe your problem more?

RMachnik
  • 3,598
  • 1
  • 34
  • 51