9

I have the "'System.OutOfMemoryException" exception for this simple code (a 10 000 * 10 000 matrix) multiplied by itself:

#time

#r "Microsoft.Office.Interop.Excel"
#r "FSharp.PowerPack.dll"

open System
open System.IO

open Microsoft.FSharp.Math
open System.Collections.Generic


let mutable Matrix1 = Matrix.create 10000 10000 0.

let matrix4 = Matrix1 * Matrix1

I have the following error: 

System.OutOfMemoryException: An exception 'System.OutOfMemoryException' has been raised
   Microsoft.FSharp.Collections.Array2DModule.ZeroCreate[T](Int32 length1, Int32 length2)
   Microsoft.FSharp.Math.DoubleImpl.mulDenseMatrixDS(DenseMatrix`1 a, DenseMatrix`1 b)
   Microsoft.FSharp.Math.SpecializedGenericImpl.mulM[a](Matrix`1 a, Matrix`1 b)
   <StartupCode$FSI_0004>.$FSI_0004.main@() dans C:\Users\XXXXXXX\documents\visual studio 2010\Projects\Library1\Library1\Module1.fs:line 92
Stop due to an error

I have therefore 2 questions:

  1. I have a 8 GB memory on my computer and according to my calculation a 10 000 * 10 000 matrix should take 381 MB [computed this way : 10 000 * 10 000 = 100 000 000 integers in the matrix => 100 000 000 * 4 bytes (integers of 32 bits) = 400 000 000 => 400 000 000 / (1024*1024) = 381 MB] so I cannot understand why there is an OutOfMemoryException

  2. More generally (it's not the case here I think), I have the impression that F# interactive registers all the data and therefore overloads the memory, do you know of a way to free all the data registered by F# interactive without exiting F#?

John Saunders
  • 160,644
  • 26
  • 247
  • 397
katter75
  • 281
  • 2
  • 9
  • 9
    You have arbitrarily much *memory* in your machine; remember, disk space is memory too. RAM chips are just a fast way to cache disk memory. You could have 8GB or .5 GB or 1000GB of physical memory; it's completely irrelevant. The memory you are running out of is *per-process virtual address space*, and that has nothing to do with how much RAM or disk space you have. – Eric Lippert Dec 21 '11 at 15:13

2 Answers2

14

In summary, fsi is a 32-bit process; at most it can hold 2GB of data. Run your test as a 64-bit Windows application; you can increase the size of the matrix, but it still has 2GB limit of .NET objects.

I correct your calculation a little bit. Matrix1 is a float matrix, so each element occupies 8 bytes in memory. The total size of Matrix1 and matrix4 in memory is at least:

2 * 10000 * 10000 * 8 = 1 600 000 000 bytes ~ 1.6 GB

(ignoring some bookkeeping parts of matrix)

So it's no surprise when fsi*32 runs out of memory in this case.

Execute the test as a 64-bit Windows process, you can create float matrices of size around 15000 but not more than that. Check out this informative article for concrete numbers with different types of matrix elements.

Community
  • 1
  • 1
pad
  • 41,040
  • 7
  • 92
  • 166
  • There's an article here that explains how to run fsi as 64 bit process: http://ig2600.blogspot.com/2010/05/making-fsharp-interpreter-fsi-run-in.html Alternatively you could compiler your F# code and by default it would run as a 64 bit process on a 64 bit OS. – Robert Dec 22 '11 at 10:52
9

The amount of physical memory on your computer is not the relevant bottleneck - see Eric Lippert's great blog post for more information.

kvb
  • 54,864
  • 2
  • 91
  • 133