Why does an asp.net web service take up so much memory?

Question

I have a .net web service (Web API 2) that takes requests to get data from a database. The returned data is not very large, just 1-3 kb usually (about 1-15 rows from the database). I set it up in IIS and ran some bulk testing on it, where it gets hit a lot for an hour or so, so it is getting several thousand requests. When I look at the w3wp.exe (IIS Worker process), the memory just keeps increasing until it gets to over 5 gb and the CPU usage gets near 100% and then eventually the web service stops working. The server has 16 gb of RAM, we just increased it from 8. Is 5 gb of memory normal? That seems like a lot, I would have assumed garbage collection would handle this better. I am not very experienced in using the performance monitor or other things to troubleshoot problems like this.

Any suggestions on what I can look at? Could there be a memory leak? Should I try having the app pool recycle when it hits a certain memory amount?

Update- we have another .net web service that does not even access a database or any external files or anything, and it is acting the same way- memory just keeps increasing. Our plan is to probably set the app pool for each to recycle every x minutes or when it gets to a certain memory amount.

Here is an example of a typical function in the API:

[Route("patient/{patientKey}/activations")]
public async Task<IHttpActionResult> GetActivations(int patientKey)
{
     try
     {
          ActivationList actList = new ActivationList();

          actList.Activations = await _actProc.GetByPatient(patientKey);

          return Ok<ActivationList>(actList);
     }
     catch (Exception ex)
     {
         return new BadRequestWithInfoResult(Translators.makeXML<ErrorReturn>(CreateError(ex)));
     }
}

public async Task<List<Activation>> GetByPatient(long patientKey)
{
     using (var dbConn = _database.CreateConnection())
     {
          List<DBActivation> lst_Activation = await dbConn.FetchAsync<DBActivation>("select * from fact.Activation where PatientKey = " + patientKey.ToString());

          List<Activation> Activations = lst_Activation.Select(x => _mapper.Map<Activation>(x)).ToList<Activation>();              

          return Activations;
     }
}

Answer 1

What's in the Activation object? If activation has many children, during the mapping, it might try to populate all the children and children of the children and on... If you are using EF lazy loading, EF will create a subquery for each child population.

Answer 2

OP answer- I found some offending code (and no, it's not normal for a web service like this to use more than 5 gb of memory). A while back, when trying to do my own namespaces in the xml returned by the web service, I had added the class CustomNamespaceXmlFormatter specified in this post under the answer by @Konamiman: Remove namespace in xml The commenter below it mentions a memory leak issue. While the ANTS Memory Profiler never showed my web service generating multiple dynamic assemblies, I nevertheless updated the code to use something similar to a singleton pattern for creating the instances of XmlSerializer like below and now my memory usage is way under control (and actually goes down when it's done processing requests!).

public class CustomNamespaceXmlFormatter : XmlMediaTypeFormatter
{
    private readonly string defaultRootNamespace;

    public CustomNamespaceXmlFormatter() : this(string.Empty)
    {
    }

    public CustomNamespaceXmlFormatter(string defaultRootNamespace)
    {
        this.defaultRootNamespace = defaultRootNamespace;
    }

    public override Task WriteToStreamAsync(Type type, object value, Stream writeStream, HttpContent content, TransportContext transportContext)
    {
        if (type == typeof(String))
        {
            //If all we want to do is return a string, just send to output as <string>value</string>
            return base.WriteToStreamAsync(type, value, writeStream, content, transportContext);
        }
        else
        {
            XmlRootAttribute xmlRootAttribute = (XmlRootAttribute)type.GetCustomAttributes(typeof(XmlRootAttribute), true)[0];
            if (xmlRootAttribute == null)
                xmlRootAttribute = new XmlRootAttribute(type.Name)
                {
                    Namespace = defaultRootNamespace
                };
            else if (xmlRootAttribute.Namespace == null)
                xmlRootAttribute = new XmlRootAttribute(xmlRootAttribute.ElementName)
                {
                    Namespace = defaultRootNamespace
                };

            var xns = new XmlSerializerNamespaces();
            xns.Add(string.Empty, xmlRootAttribute.Namespace);

            return Task.Factory.StartNew(() =>
            {
                //var serializer = new XmlSerializer(type, xmlRootAttribute); **OLD CODE**
                var serializer = XmlSerializerInstance.GetSerializer(type, xmlRootAttribute);
                serializer.Serialize(writeStream, value, xns);                    
            });
        }
    }
}

public static class XmlSerializerInstance
{
    public static object _lock = new object();
    public static Dictionary<string, XmlSerializer> _serializers = new Dictionary<string, XmlSerializer>();
    public static XmlSerializer GetSerializer(Type type, XmlRootAttribute xra)
    {
        lock (_lock)
        {
            var key = $"{type}|{xra}";
            if (!_serializers.TryGetValue(key, out XmlSerializer serializer))
            {
                if (type != null && xra != null)
                {
                    serializer = new XmlSerializer(type, xra);
                }

                _serializers.Add(key, serializer);
            }

            return serializer;
        }
    }
}

Answer 3

Seems to be a deadlock because of not using ConfigureAwait in the library,

[Route("patient/{patientKey}/activations")]
public async Task<IHttpActionResult> GetActivations(int patientKey)
{
     try
     {
          ActivationList actList = new ActivationList();

          actList.Activations = await _actProc.GetByPatient(patientKey).ConfigureAwait(false);

          return Ok<ActivationList>(actList);
     }
     catch (Exception ex)
     {
         return new BadRequestWithInfoResult(Translators.makeXML<ErrorReturn>(CreateError(ex)));
     }
}

public async Task<List<Activation>> GetByPatient(long patientKey)
{
     using (var dbConn = _database.CreateConnection())
     {
          List<DBActivation> lst_Activation = await dbConn.FetchAsync<DBActivation>("select * from fact.Activation where PatientKey = " + patientKey.ToString()).ConfigureAwait(false);

          List<Activation> Activations = lst_Activation.Select(x => _mapper.Map<Activation>(x)).ToList<Activation>();              

          return Activations;
     }
}