0

I am working on an application which requires to fetch entities from one system (Contacts from Dynamics CRM). The app does some process on those contacts, basically convert to compatible payload for other system and then sync it back.

The Dynamics CRM has over 8000 contacts so no brainer, I went ahead with pagination approach.. The code looks something like this - 

string nextPageUrl = string.Empty;
bool nextPageAvailable = true;
int totalContactsCount = 0;
bool firstCall = true;
JArray totalContacts = new JArray();
while (nextPageAvailable)
{
    string contacts = string.Empty;
    if (string.IsNullOrEmpty(nextPageUrl) && firstCall)
    {
        contacts = await _genericService.GetEntity("contacts", pageSize: 100);
        firstCall = false;
    }
    else
    {
       contacts = await _genericService.GetEntityFromPaginationUrl(nextPageUrl, pageSize: 100);
    }
    var contactsObj = JObject.Parse(contacts);
    var contactArr = (JArray)contactsObj["value"];
    var contactsLength = contactArr.Count;

    foreach(var contactObject in contactArr)
    {
       totalContacts.Add(contactObject);
    }
    var contactsCt = totalContacts.Count;

    nextPageUrl = contactsObj["@odata.nextLink"].IsNullOrEmpty() ? string.Empty : contactsObj["@odata.nextLink"].ToString();

    if(string.IsNullOrEmpty(nextPageUrl) && !firstCall)
    {
       nextPageAvailable = false;
    }

    Console.WriteLine(nextPageUrl);
 }

This works perfectly but till I get 8000 contacts (100 at a time), the memory utilization is almost over 370mb. I am sure there is a better way to achieve this but can not think of any.

The memory problem will get more serious when I reiterate through totalContacts which will be having over 8000 contacts and process them.

Since the contacts coming from Dynamics CRM may have newly added properties, I can not strongly type the objects with models. This is why I have decided to go with JArray. Is there any better alternative to achieve this?

Thanks in advance.

Rahul Patil
  • 5,656
  • 6
  • 37
  • 65
  • 1) How large is each `contacts` string? is it ever large enough to go on the [large objects heap](https://stackoverflow.com/questions/8951836/why-large-object-heap-and-why-do-we-care)? 2) For comparison, if you create strongly typed models using the currently returned properties, what is the memory use? – dbc Feb 17 '17 at 14:33
  • Also, can you explain why you are loading all these contacts into memory at once? Is sequential processing of chunks not possible? – dbc Feb 17 '17 at 15:12

0 Answers0