System.OutOfMemoryException for a tail recursive function

Question

I have isolated the problematic code to this function (that uses ASP.NET's Membership class):

let dbctx = DBSchema.GetDataContext()
let rec h1 (is2_ : int) (ie2_ : int) : unit =
    match is2_ >= ie2_ with
    | true ->
        let st2 = query {
            for row in dbctx.Tbl_Students do
            where (row.Id = is2_)
            head}
        let l2 =
            Membership.FindUsersByEmail (st2.Email_address)
            |> Seq.cast<_>
            |> Seq.length
        match l2 >= 1 with
        | true -> 
            ()
        | false ->
            Membership.CreateUser (st2.Email_address, password, st2.Email_address)
            |> ignore
        h1 (is2_ - 1) ie2_
    | false ->
        ()

I am getting a System.OutOfMemoryException after exactly 5626 iterations of h1. But my system's memory consumption is only at 20 percent. (I have a very powerful, 16 GB machine.)

Why should the above function overflow the stack? Is it not written tail recursively?

Thanks in advance for your help.

Are you running in Debug mode? If so, tail calls are disabled. Try your code in Release mode. — Daniel, Jan 14 '13 at 16:58
I would rewrite the last part using `Seq.isEmpty`. There is no need to enumerate the whole sequence. — pad, Jan 14 '13 at 17:06
@Daniel Is that so? I have numerous other tail recursive functions in the project, which traverse greater depths without throwing an error, in the same Debug mode. Still, I'll check if that is the issue. Thanks for the suggestion. — Shredderroy, Jan 14 '13 at 17:20
@pad, thanks for the suggestion. I have made the change (in my actual code). — Shredderroy, Jan 14 '13 at 17:22
Is it possible that it is the data context which is being reused and accumulating data. I would expect this line to be - let dbctx() = DBSchema.GetDataContext() instead, to create a new data context each time. — hocho, Jan 16 '13 at 16:30
@Lucifure I didn't realise that reusing a data context was a problem (even though I knew to avoid it in Entity Framework in C#). Even so, the table only has 10,000 rows, and each row only has about 2 kB of data, so I would not have imagined that data accumulation could have been the problem. For what it is worth, the problem still persists, but since the solution needs to run on the whole table only once, and then rum incrementally each time a new user is created, I got around the issue by looping the function call and using 500 rows at a time. — Shredderroy, Jan 16 '13 at 17:26

score 5 · Accepted Answer · edited May 23 '17 at 12:15

I don't think this is a tail-recursion issue -- if so, you'd be getting a StackOverflowException instead of an OutOfMemoryException. Note that even though you have 16GB of memory in your machine, the process your program is executing in may be limited to a smaller amount of memory. IIRC, it's 3GB for some combinations of .NET framework version and OS version -- this would explain why the process is crashing when you reach ~20% memory usage (20% of 16GB = 3.2GB).

I don't know how much it'll help, but you can simplify your code to avoid creating some unnecessary sequences:

let dbctx = DBSchema.GetDataContext()
let rec h1 (is2_ : int) (ie2_ : int) : unit =
    if is2_ >= ie2_ then
        let st2 = query {
            for row in dbctx.Tbl_Students do
            where (row.Id = is2_)
            head }
        let existingUsers = Membership.FindUsersByEmail st2.Email_address
        if existingUsers.Count < 1 then
            Membership.CreateUser (st2.Email_address, password, st2.Email_address)
            |> ignore

        h1 (is2_ - 1) ie2_

EDIT : Here's a link to a previous question with details about the CLR memory limitations for some versions of the .NET framework and OS versions: Is there a memory limit for a single .NET process

score 5 · Answer 2 · answered Jan 14 '13 at 17:31

OutOfMemoryException usually doesn't have anything to do with the amount of RAM you have. You get it at ~3 GB most likely because your code runs as a 32-bit process. But switching it to 64 bit will only solve your issue if you actually need that much memory and the exception is not caused by some bug.

System.OutOfMemoryException for a tail recursive function

2 Answers2