Https calls are not connecting to server

Question

I am working on Windows Service in visual studio 2017. In the rest api's call, getting exceptions while debugging code. Sometimes first 2 3 calls working after that getting exceptions.

System.Net.WebException: 'The remote server returned an error: (503) Server Unavailable.'

The remote server returned an error: (429)

Unable to connect to the remote server

When calling same api's from Postman, getting response successfully.

This is my code

private void timer1_Tick(object sender, ElapsedEventArgs e)
{
    WriteToFile("timer1_Tick method called..");
try
{
    string jsonString = "";
    string jsonstring2 = "";
    string prodfetchurl = HOST;
    var req = WebRequest.Create(prodfetchurl) as HttpWebRequest;
    req.Method = "GET";
    InitializeRequest(req);
    req.Accept = MIME_TYPE;
    //System.Threading.Thread.Sleep(5000);
    var response = (HttpWebResponse)req.GetResponse();
    WriteToFile("First service called...");
    if (response.StatusCode == HttpStatusCode.OK)
    {
        Stream responseStream = response.GetResponseStream();
        StreamReader responseReader = new StreamReader(responseStream);
        jsonString = responseReader.ReadToEnd();
    }
    var deserialsseobj = JsonConvert.DeserializeObject<ProductList>(jsonString).Products.Where(i => i.Failed > 0).ToList();
    foreach (var a in deserialsseobj)
    {
        var pid = a.ID;
        string url = FailedDevicesUrl + pid.Value + "/failed";
        var req2 = WebRequest.Create(url) as HttpWebRequest;
        req2.Method = "GET";
        InitializeRequest(req2);

        req2.Timeout = 300000;
        req2.Accept = MIME_TYPE;
        var response1 = (HttpWebResponse)req2.GetResponse();
        Stream responsestream2 = response1.GetResponseStream();
        WriteToFile("Second service called...");
        if (response1.StatusCode == HttpStatusCode.OK)
        {
            StreamReader responsereader1 = new StreamReader(responsestream2);
            jsonstring2 = responsereader1.ReadToEnd();
        }

        var output = JsonConvert.DeserializeObject<List<FailedDeviceList>>(jsonstring2);  // Will get List of the Failed devices
        List<int> deviceids = new List<int>();
        Reprocessdata reproc = new Reprocessdata();
        Reprocessdata.DeviceId rprod = new Reprocessdata.DeviceId();

        reproc.ForceFlag = true;
        reproc.ProductID = pid.Value;
        foreach (var dd in output)
        {
            rprod.ID = dd.DeviceId;
            reproc.DeviceIds.Add(rprod);
        }

        // Reprocess the Product in Devices
        var req3 = WebRequest.Create(ReprocessUrl) as HttpWebRequest;
        req3.Method = "POST";
        InitializeRequest(req3);
        req3.Accept = MIME_TYPE;
        req3.Timeout = 300000;
        req3.ContentType = "application/json";
        using (StreamWriter writer = new StreamWriter(req3.GetRequestStream()))
        {
            string json = new JavaScriptSerializer().Serialize(reproc);

            writer.Write(json);
            writer.Close();
        }
        System.Threading.Thread.Sleep(5000);
        var response5 = (HttpWebResponse)req3.GetResponse();
        WriteToFile("Third service called...");
        if (response5.StatusCode == HttpStatusCode.OK)
        {
            string result;
            using (StreamReader rdr = new StreamReader(response5.GetResponseStream()))
            {
                result = rdr.ReadToEnd();
            }
        }
    }
    response.Close();
}
catch (Exception ex)
{
    WriteToFile("Simple Service Error on: {0} " + ex.Message + ex.StackTrace);
}
}

Methods used in above code

protected override void OnStart(string[] args)
{
    base.OnStart(args);
    timer1 = new System.Timers.Timer();
    timer1.Interval = 60000; //every 1 min
    timer1.Elapsed += new System.Timers.ElapsedEventHandler(timer1_Tick);
    timer1.Enabled = true;
    WriteToFile("Service has started..");
}

public void InitializeRequest(HttpWebRequest request)
{
    request.Headers.Add("aw-tenant-code", API_TENANT_CODE);
    request.Credentials = new NetworkCredential(USER_NAME, PASSWORD);
    request.KeepAlive = false;
    request.AddRange(1024);
}

When I contacted service provide they said everything fine from there side. Is this my code is buggy or windows service not reliable? How can I fix this issue?

Note: All APIS are working fine from Angular application using Visual Studio Code. It means my code is not working.

Edit1: Three below services I am using from this document of VMware.

private const string HOST = "https:host/api/mdm/products/search?";
private const string FailedDevicesUrl = "https:host/api/mdm/products/";
private const string ReprocessUrl = "https:host/api/mdm/products/reprocessProduct";

Answer 1

The inner error of that 503 is:

The remote server returned an error: (429)

HTTP 429 indicates too many requests. Maybe your upstream server can’t process all requests sent.

This can happen when you reached rate limiting / throttling value if you’re calling a third party API.

UPDATE

As per page 28 in the API docs, you could configure throttling when creating a new API. Check if the throttling is too small or maybe turn off the throttling and see if that could fix the error?

Answer 2

Response http code 429 indicates that you sending too many requests on target web service.

This means service you trying to send requests has a policies that blocks some requests by request-per-time limit.

Also I admit that external service can be manually configured to throw 403 code in specific cases that you can't know about. If that, this information can be explained in external service documentation... or not :)

What you can do with this?

Fit in limitations
You can make detailed research what limits target webservice has and set up your code to fit in this limitations. For example if service has limitation for receiving only one request per 10 minutes - you must set up your timer to send one request each 10 or more minutes. If documentation not provide such information - you can test it manually by finding some patterns with external service responses.

Use proxy
Every limitation policy based on information about requests senders. Usually this information consists of IP address of sender only. This means if you send 2 requests from two different IP addresses - limitation policy will perceive that like 2 different computers sending these requests. So you can find/buy/rent some proxy IP addresses and send requests through there on target web server.

How to connect through proxy in C# using WebRequest you can see in this answer.

Negotiate with external service provider
If you have possibility to communicate with external service developers or help center, you can ask their to reduce limitations for your IP address (if it static) or provide some mechanisms to avoid limitation policy for you. If for some reason they cannot provide this opportunity, at least you can ask detailed information about limitations.

Repetition mechanism
Some times 503 error code that is outer exception you received may be caused by service unavailable. It means that server can be under maintenance or temporary overloaded. So you can write repetition mechanism to make continious sending requests to server until it'll be accessible.

Polly library may help you with repetition mechanism creation