0

I'm trying to set up a basic web service (.NET 4.5.2 hosted on IIS 7.5) to query some data in Dynamics CRM using the SDK (we're using CRM 2015 online). Depending on the query there could be a lot of records to return, for example querying the Account entity I might expect to get back as much as 250000 records. I'm using paging to get the results and stuff them into a final EntityCollection object so I can work on the full data set once all the records are returned. The problem is I'm getting an OutOfMemory exception usually before it finishes querying all pages. Here's a code snippet showing the While loop I'm working with:

While lblnMoreRecords
   lEC = _CRMHelper.OrgService.RetrieveMultiple(New FetchExpression(lstrXml))
   lblnMoreRecords = lEC.MoreRecords

   If lEC.Entities.Count > 0 Then
      pecEntities.Entities.AddRange(lEC.Entities)
      lintPage += 1
      lstrXml = SetFetchXMLPaging(pstrFetchXML, lEC.PagingCookie, lintPage, 200)
   Else
      lblnMoreRecords = False
   End If

   lEC = Nothing
End While

In this snippet I'm using FetchXML to get the results, although I have also tried the same sort of loop with a QueryExpression object and paging. I can run this process and watch it's memory usage as it's running, the memory consumption comes in at the line lEC = _CRMHelper.OrgService.RetrieveMultiple(New FetchExpression(lstrXml)), where it is doing the actual query and returning the results from CRM. The process will occasionally free up some memory but overall it increases until I get the exception. Besides trying a QueryExpression, I also tried reducing the number of records per page from 1000 to 200, hoping that by using less memory for each iteration would mean the OS could more easily allocate free blocks of memory to the process. That helped but I still get the exception before all results are returned.

One more thing I tried was to run this query in a basic Windows App rather than a web service. The windows app seemed to be much better at managing memory, it got to a certain point and stayed steady, easily finishing the query and getting all the results.

Any suggestions on dealing with this issue in a web service/IIS would be greatly appreciated, or does anyone know of a better way to query CRM via their SDK to get large sets of data? Thanks.

Matt
  • 4,656
  • 1
  • 22
  • 32
AK3800
  • 2,138
  • 3
  • 23
  • 28
  • Why do you need to keep al records in memory? Can't you store the data retrieved in a temporary file? – Henk van Boeijen Nov 17 '15 at 22:55
  • I could try that, but I'm not sure it will help. To answer your question, I need to eventually get all the records into a collection to send back to the client calling the web service. It doesn't seem like the `EntityCollection` object where I'm storing the retrieved records is an issue at all. Where I'm seeing the big memory grab is in the `RetrieveMultiple` or `Execute` methods that are retrieving a page of records from CRM, so it seems to be the SDK itself that is the problem. I'm wondering if I can handle the SDK object differently to make sure it frees up memory after each retrieval? – AK3800 Nov 18 '15 at 02:02
  • Sounds like deserialization generates a lot of garbage. You can call the GC in between. Nevertheless, it seems to me it's not a Dynamics CRM-specific issue you're dealing with. What are your machine specs? – Henk van Boeijen Nov 18 '15 at 08:02
  • Do you have an example of explicitly calling the garbage collector, I've looked for that too and haven't found any examples. I'm running a Core i7 with 12GB RAM. – AK3800 Nov 18 '15 at 20:13
  • It's `GC.Collect();`. With 12 GB RAM I'm wondering what is going on on your computer. Are you retrieving a large amount of columns, perhaps? – Henk van Boeijen Nov 18 '15 at 20:30
  • I'm not using up all the memory on my machine, I had about 4GB free at the time I was running this process and the OutOfMemory exception occurred. The exception just means that the process didn't get any more free blocks of memory available to it. That's why I tried reducing the number of records so it would work with smaller blocks of memory, thus increasing the chance it would have free contiguous blocks to use up. That helped but still it seems like the retrieval is using WAY more memory than it should and is not cleaning up after itself. – AK3800 Nov 19 '15 at 00:00
  • Even under x64 there is a 2 GB limit for a single object. See [Memory limitations in a 64-bit .Net application](http://stackoverflow.com/q/6107322/205233). – Filburt Nov 19 '15 at 19:00

0 Answers0