0

I'm writing a Web Application with a Java back-end running on a Tomcat server and a JavaScript client.

In the backend I have to process a large int[][][] array, which holds the information of a CT-Scan. Size is approx. 1024x1024x200.

I want to load this array into memory only when it's needed to process new data like image slices, and store it in some kind of database for the remaining time.

Things I tried so far:

  • Using JDBM3 to store a String, int[][][] Hashmap, runs into out of memory error

  • Serializing object and save it into PostgreSQL-DB using bytea[] data type, stores correctly but is getting memory error while loading again.

So my first question is, how can I save such a big array (which db, method)? It should load fast, and there should be some kind of multi-user access security because multiple users will be able to use front-end and therefore load the int[][][] into the back-end. The database should have a non-commercial license eg. GPL, MIT, Apache...

Second question, I know I could save the array serialized in the file system and keep the link in the db, but is the access safe for multiple users?

Flexo
  • 87,323
  • 22
  • 191
  • 272

3 Answers3

2

If you have enough RAM on the client machines, you could start by simply increasing the size of the JVM heap. This way, you should be able to create larger arrays without running into 'out of memory' errors.

You'll need at least approximately 800 Mb to play with (1024 x 1024 x 200 x 32 bits) for the array alone.

ChristopheD
  • 112,638
  • 29
  • 165
  • 179
  • Hi, the heapspace on the server is big enough to process, but the problem is, if 10 users are using the webapplication, 10 cubes would be in the memory the whole time. I want to load the cube out of an db into memory only when i need it, so that the cubes are not in the memory the whole time. –  May 29 '12 at 19:50
  • @Xandman: have you considered writing the files directly to disk (and storing references to it in the database)? A related thread: http://stackoverflow.com/questions/4358875/fastest-way-to-write-an-array-of-integers-to-a-file-in-java – ChristopheD May 29 '12 at 19:55
  • Yes I Considered storing directly on disk, but as I asked in my second question, what happens if more than one user are trying to access the file at the same time ? –  May 29 '12 at 20:13
1

I think a MemoryMappedFile was born to handle exactly stuff like this. It offers you an array-like view of a file on disk, with random access for read and write. All you have to do is develop a scheme that lays out an int[][][] over a byte[] and that shouldn't be a problem. If you do it this way, you'd never have to hold the whole array in memory, but create only the slices that you are actually using. Even if you need to iterate over all the slices, you can instantiate only a single slice at a time.

Marko Topolnik
  • 195,646
  • 29
  • 319
  • 436
0

If its a CAT-Scan, are the pixels 256 color grayscale? if so, you can save a significant amount of memory by storing the data as a byte-array rather than an int array. If it is 64K grayscale, use a short rather than an int.

JohnnyO
  • 3,018
  • 18
  • 30