2

I have implemented a scheduler that sends request to a third party endpoint. After receiving the response, my local database is updated with the response. Currently, I am sending more than 50K requests(1K requests every 10 minutes) and process the response. The issue is sometimes the third party server doesn't responds or the request time out. In this case I get an exception and the thread is aborted without processing the remaining requests. What I need is not to abort the thread and move on with the next record so that the missed out record is processed in another batch. Here is the code I am using.

public class ScheduledAPIJob : IJob
{
    public Task Execute(IJobExecutionContext context)
    {
        Task taskAPI = Task.Factory.StartNew(() => ProcessAPI());
        return taskAPI;
    }
    void ProcessAPI()
    {
        //Error logging object
        SchedulerLogWriter lw = new SchedulerLogWriter("Logs\\Scheduler");

        List<WeatherData> list = new List<WeatherData>();

        APIQueueBAL objBal = new APIQueueBAL();

        //List of endpoints to hit.
        var APIQueue = objBal.QueuedAPIs();

        foreach (var item in APIQueue)
        {
            try
            {
                var endpoint = item.FunctionParameters;
                HttpRequestHelper objRequestHelper = new HttpRequestHelper();
                
                //Response from API
                var response = objRequestHelper.GetAPIResponse(endpoint);
                
                ////Update local database.
                if (response.StatusCode == System.Net.HttpStatusCode.OK)
                {
                    list = JsonConvert.DeserializeObject<List<WeatherData>>(response.Content.ReadAsStringAsync().Result);
                    objBal.ProcessWeatherData(item, list);
                }
            }
            catch (Exception ex)
            {
                lw.WriteLog(ex.Message);
                lw.WriteLog(Convert.ToString(ex.InnerException));
                lw.WriteLog(ex.StackTrace);
            }
        }
    }
}

public class HttpRequestHelper
{
    public HttpResponseMessage GetAPIResponse(string apiEndpoint)
    {
        using (var client = new HttpClient())
        {
            var getTask = client.GetAsync(apiEndpoint);
            getTask.Wait();
            return getTask.Result;
        }
    }
}
d51
  • 316
  • 1
  • 6
  • 23
astm1982
  • 145
  • 13
  • Your try catch means the loop should continue to the next entry when an exception occurs. Are you saying that isn't happening? – mjwills Aug 18 '20 at 06:52
  • 3
    `using (var client = new HttpClient())` You should `new` up one of these - not one per request. – mjwills Aug 18 '20 at 06:53
  • 4
    1) Don't create and dispose of lots of HttpClient's. Keep them around so that TCP connections can be reused. 2) Why not convert everything to async methods? – Jeremy Lakeman Aug 18 '20 at 06:53
  • I can't see `Thread`s in your code. I am seeing `Task`s. What do you mean by "thread aborts"? Do you get a [`ThreadAbortException`](https://learn.microsoft.com/en-us/dotnet/api/system.threading.threadabortexception)? – Theodor Zoulias Aug 18 '20 at 06:58
  • And this does not work? you have a try-catch in your foreach loop, so if something happens it will just move to the next item. We need more information. – sommmen Aug 18 '20 at 07:23
  • @mjwills Yes, I am getting ThreadAbortException. – astm1982 Aug 18 '20 at 07:27
  • 1
    Please share a [mcve]. – mjwills Aug 18 '20 at 07:43
  • Where is `IJob` defined? – mjwills Aug 18 '20 at 07:44
  • 1
    Since there are no threads in your code, nor any code that obviously aborts threads, it is unfortunately going to be impossible to answer your question. I would sprinkle logging statements liberally throughout your code so that you can isolate where the abort is being done. It has to be in some code you're calling, that you haven't shown us. – Lasse V. Karlsen Aug 18 '20 at 07:53
  • 1
    @LasseV.Karlsen [Look](https://stackoverflow.com/a/33724034/12888024). That's easy to solve that. – aepot Aug 18 '20 at 08:14
  • 1
    @LasseV.Karlsen one more [useful link](https://stackoverflow.com/a/27830926/12888024) – aepot Aug 18 '20 at 08:19
  • @astm1982 - Thread abort exceptions are dangerous. The aborted thread can leave the run-time in a state that can causes all code to become unreliable. You should only ever call `.Abort()` when you are trying to forcibly close your app. If a third-party library is doing that then the only safe option is to close down as soon as you catch the exception unless you can defensively programme against the exception in the first place. – Enigmativity Aug 18 '20 at 09:11

0 Answers0