.Net framework 4.0 call REST API sometimes is slow

Question

I have a WCF service, hosted on premisses, .Net Framework 3.5. Internally, in one operation that is exposes, it calls an external API that is hosted in Azure which returns a reply in JSON format.

This third party API is quite fast, usually it responds below 300ms, most of the times even below 100ms. The size of responses is less than 3KB.

What happens is that now & then I find that this WCF operation is quite slower than usual, and drilling down with some tooling (in this case, NewRelic), I find something like this:

STREAM/Get 11s

And it doesn't match with the times of API that is being called.

The piece of code that interacts with this API:

private List<Location> GetStuff()
    {
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | (SecurityProtocolType)768 | (SecurityProtocolType)3072;

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://someendpoint");
        request.ContentType = "application/json";
        request.Headers["Authorization"] = "Bearer bla bla bla";

        using (var response = request.GetResponse())
        {
            NewtonsoftJsonSerializer serializer = new NewtonsoftJsonSerializer();
            
            // Deserialize response
            return serializer.Deserialize<List<Location>>(response.GetResponseStream());
        }        
    }

    public class Location
    {
        public string LocationId { get; set; }
        public int LocationTypeId { get; set; }
        public string Name { get; set; }
        public string Code { get; set; }
        public bool? Active { get; set; }
        public string DisplayName { get; set; }
        public List<Location> Children { get; set; }
    }

My question is: is the part of deserialization unefficient some how? Should I rewrite it in another manner? I'm not 100% sure, but this is the only culprit I find.

Other than this, could it be due to the fact that I'm calling an Azure API from an on premisses project?

Thank you