0

What are the Memory restrictions of the ILNumerics Library ? A hdf5 Dataset of Byte [27 Mio, 24] (108 MB) can read as Byte

using (var file = new H5File(@"MP2.h5"))
  { var data = file.First<H5Dataset>("Spectrum").Get<Byte>();

The Taskmanager show then 650 MB used for the Process.

If I set Get to float an Exception is thrown:

System.OutOfMemoryException: Die Arraydimensionen haben den unterstützten Bereich überschritten.
 bei ILNumerics.Misc.ILMemoryPoolInternal`1.New(Int64 length)
 bei ILNumerics.IO.HDF5.H5Dataset.Get[T](ILBaseArray[] range)
 bei WindowsFormsApplication1.Plotting_Form1..ctor() in

I have Win7x64 with 16GB and use AnyCPU. So an 500 MB Array should be possible.

Matlabx64 with 300 MB RAM used after start can read the data as byte (940 MB), single (2.6 GB RAM used) or double (5.3 GB).

Pete
  • 673
  • 5
  • 8
  • This is by design: http://stackoverflow.com/questions/2338778/what-is-the-maximum-length-of-an-array-in-net-on-64-bit-windows. You cannot read such huge arrays as one piece into memory but you can read subarrays from it. Just define a subarray definition like `Get("1000:3000;5000:7000")`. When working with big data you should adapt to thinking in chunks anyway. Otherwise there will always be a wall to be hit at some point... – Haymo Kutschbach Aug 26 '14 at 22:39
  • Also, have you checked which bitrate your program runs at? "AnyCPU" could mean "x86"? (Not that x64 would help, though) – Haymo Kutschbach Aug 26 '14 at 22:42
  • AnyCPU ist started as x64 process. – Pete Aug 27 '14 at 08:05
  • Ok. System.Array float[27 000 000,24] exceeds the 2GB maximum single object restriction in the .NET CLR. There is a workaround.. Of course later the data must read in chunks. It should also work on 32 bit Systems. – Pete Aug 27 '14 at 08:11

0 Answers0