RavenDb Ver 3.0 - "Query Failed" with "OutOfMemoryException" Message on Large data

Question

We are developing new application on top of RavenDb Ver 3.0 as Data storage.

On performance tests we found an issue with our system.

The issue is that when we run a query like below, to take all the results, if the total results become more than sth (like >2000 document or sometimes even less)

the "Query Failed" with "OutOfMemoryException" Message appears.

public ICollection<T> Find<T>(Expression<Func<T, bool>> predicate)
{
    var spendTime01 = Stopwatch.StartNew();
    var list = new List<T>();
    var power = 2000;

    RavenQueryStatistics statistics;
    using (DocumentSession)
    {
         list.AddRange(DocumentSession.Query<T>().Statistics(out statistics)
                                                 .Where(predicate).Take(power));
    }

    if (statistics.TotalResults > power)
    {
       var toTake = statistics.TotalResults - power;
       var taken = power;

       while (toTake > 0)
       {
            using (DocumentSession)
            {
               list.AddRange(
                   DocumentSession.Query<T>()
                       .Where(predicate)
                       .Skip(taken)
                       .Take(toTake > power ? power : toTake));
               toTake -= power;
               taken += power;
            }
       }
}

//using (DocumentSession)
//{
//    var query = DocumentSession.Query<T>("Activities/All").Where(predicate);
//    using (var enumerator = DocumentSession.Advanced.Stream(query))
//    {
//        while (enumerator.MoveNext())
//        {
//            list.Add(enumerator.Current.Document);
//        }
//    }
//}

spendTime01.Stop();
Debug.WriteLine($"Raven Find Predicate Elapsed Time: {spendTime01.Elapsed}");
return list;
}

I tried to use index and switch to Advanced.Stream(query) but this command is running very slow and for 4000 document takes 20 Sec to enumerate the result and add it to list.

I read all the blogs and answered around the net about this and yet I don't have any POV about the problem and also I'm completely worry about what will happen if the total results become more than hundred of thousands.

How big are each document? This is the equivalent of a SELECT * which has been the cause of many OutOfMemory exceptions... Why do you need to query for everything? — Jens Pettersson, Jun 13 '16 at 20:16
@JensPettersson each document is about 40kb. The reason of selecting all is to extract report infomtaion which is depend on every document members. — Xaimaran, Jun 14 '16 at 06:20

score 1 · Accepted Answer · answered Jun 14 '16 at 07:25

You are likely running in 32 bits mode, and you need to run in 64 bits.

As an aside, you are explicitly doing very bad things there with making many calls to the server to fetch large amount of data. Use Streaming if you need that much data, but be aware that there is very little to do with large record sets in most situations

score 0 · Answer 2 · edited May 23 '17 at 11:52

0

To complete Mr.Ayende Rahien answer I should mention the solution link to change the IIS Express to run as a 64 bit process.

Why can't I set asp.net mvc 4 project to be x64

edited May 23 '17 at 11:52

Community

1
1

answered Jun 14 '16 at 09:28

Xaimaran

403
4
17

RavenDb Ver 3.0 - "Query Failed" with "OutOfMemoryException" Message on Large data

2 Answers2