3

I've seen a fair few posts regarding this issue but non that seem to fit my criteria.

Having said that, this is the first one that I've built that "needs" to perform a tremendous amount of logic. All works great for a couple days but after that logic stops working with no log in the event viewer nor firing of an email (exceptions).

I'm wondering if my logic is not correct... Looking for advice and pointers.

public partial class QuayService : ServiceBase
{
    private System.Timers.Timer m_mainTimer;
    private bool m_timerTaskSuccess;
    private Email _email;
    public QuayService()
    {
        InitializeComponent();
        _email = new Email();
    }

    protected override void OnStart(string[] args)
    {
        try
        {
            // Create and start a timer.
            m_mainTimer = new System.Timers.Timer();
            m_mainTimer.Interval = 5000;   // every 5 seconds
            m_mainTimer.Elapsed += m_mainTimer_Elapsed;
            m_mainTimer.AutoReset = false;  // makes it fire only once
            m_mainTimer.Start(); // Start

            m_timerTaskSuccess = false;

            _email.SendEmail("Quay", "(Shopify)Quay Service Started",
                "(Shopify)Quay Service Successfuly Started");

            EventLog.WriteEntry("(Shopify)Quay Service Started...");
        }
        catch (Exception ex)
        {
            // Log/Send Email
            _email.SendEmail("Quay", "Error starting (Shopify)Quay Service", ex.Message + " " + 
                ex.InnerException.Message);

            EventLog.WriteEntry("Error starting (Shopify)Quay service and timer..." + ex.Message + " " +
                ex.InnerException.Message);
        }
    }

    protected override void OnStop()
    {
        try
        {
            // Service stopped. Also stop the timer.
            m_mainTimer.Stop();
            m_mainTimer.Dispose();
            m_mainTimer = null;

            _email.SendEmail("Quay", "(Shopify)Quay Service stopped",
                "(Shopify)Quay Service Successfuly Stopped");

            EventLog.WriteEntry("(Shopify)Quay Service stopped...");
        }
        catch (Exception ex)
        {
            _email.SendEmail("Quay", "Error stopping (Shopify)Quay Service", ex.Message + " " +
                ex.InnerException.Message);

            // Log/Send Email
            EventLog.WriteEntry("Error stopping (Shopify)Quay timer and service..." + ex.Message + " " +
                ex.InnerException.Message);
        }
    }

    void m_mainTimer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
    {
        try
        {
            var orderUoW = new OrderUoW();
            orderUoW.Create();

            m_timerTaskSuccess = true;
        }
        catch (Exception ex)
        {
            //Error with timer elapsed
            m_timerTaskSuccess = false;

            _email.SendEmail("Quay", "Error creating (Shopify)Quay order(s)", ex.Message + " " +
                ex.InnerException.Message);

            EventLog.WriteEntry("Error creating (Shopify)Quay order(Time elapsed event)..." + ex.Message
                + " " + ex.InnerException.Message);
        }
        finally
        {
            if (m_timerTaskSuccess)
            {
                m_mainTimer.Start();
            }
        }
    }
}

To get to this point, I "googled" and "SO'ed" to find the best use of timer... I also have a naïve test layer that reproduces what's in the service, I can run through that fine without any exceptions but this one is driving me nuts.

Any help truly appreciated!

//EDIT

A little explanation on:

var orderUoW = new OrderUoW(); 
orderUoW.Create();

OrderUoW is a class in my service that inherits from BatchOrder which is responsible for many actions including connecting to two databases and polling of the shopify API... OrderUow is purely to decouple from the business layer but giving access to "x" methods.

All works flawlessly for two days but just seems to halt. I'm not hitting request limits within shopify... so at the moment, I'm at a loss.

Tez Wingfield
  • 2,129
  • 5
  • 26
  • 46
  • 1
    You should have a try/catch block around `_email.SendEmail` as it can fail too. – Wagner DosAnjos Jan 12 '17 at 16:34
  • Duly noted, thank you! – Tez Wingfield Jan 12 '17 at 16:34
  • 1
    I wonder if this is possibly an [uncatchable exception](http://stackoverflow.com/a/4759831/43846) – stuartd Jan 12 '17 at 16:35
  • @stuartd cheers, I'll check that out. – Tez Wingfield Jan 12 '17 at 16:37
  • Logically it all looks ok except that any unhandled exception (as pointed out by @wdosanjos) will cause issue. You might want to take out that piece of exception handling in some DRY blocks (but that's another issue). Can you give us more detail about the `OrderUoW` object? Seems like it does something every 5 seconds so is a worth candidate for investigation... – Imran Saeed Jan 12 '17 at 16:39
  • @i.net please see edits. – Tez Wingfield Jan 12 '17 at 16:49
  • 1
    Then I believe you need to add more logging and see what happens in executions just before the crash. Like I said, your code looks ok so it's probably worth investigating external factors like database, shopify, email etc. Also add some system metrics to monitor if the OS is doing ok at the point of crash just incase there are resourcing issues like disk io, cpu, mem, network. Need to identify the problem to sort it and your code doesn't have any obvious problems. – Imran Saeed Jan 12 '17 at 17:22
  • @i.net much appreciated thanks for your time... I'll refactor to handle obvious exceptions and DRY. Whilst I've been testing over the last few days I have noticed that every now and again I cannot connect to a "DB" which would ultimately lead to an exception. Not sure why exceptions are not bubbling to the "OnStart" method though. – Tez Wingfield Jan 12 '17 at 17:32
  • I see you always log this `ex.InnerException.Message`. If there is no `InnerException` your exception handling is broken. – Peter Bons Jan 12 '17 at 18:18
  • @PeterBons Thanks, I've removed. Going through the process of defining an exception pattern/strategy. – Tez Wingfield Jan 12 '17 at 19:01
  • @TezWingfield, I don't think you should remove the inner exception completely. Just check for null and handle appropriately. – Imran Saeed Jan 13 '17 at 10:35
  • @i.net Yes, agreed. I'm just in a rush at the moment. I've recently found out that Timers.Timer has a bug where as it silently kills exceptions and then kills the timer thus not raising the event. I'm trying to figure out how to re-write in Threading.Timer. – Tez Wingfield Jan 13 '17 at 10:44
  • Threading.Timer class uses ThreadPool for executing event so make sure that your ThreadPool is configured optimally for this. Timer event won't be fired if thread pool is filled up by other processes in which case you won't get any events which will be frustrating to debug. Good luck! – Imran Saeed Jan 13 '17 at 11:35
  • @i.net too many issues... Solved. Please see answer. – Tez Wingfield Jan 16 '17 at 17:51

1 Answers1

3

After many moments of scratching my head and extreme annoyance with the MS Timer Bug.

Not only did it swallow exceptions, the timer did not raise the event! My service was hanging, which was particularly difficult because it also swallowed exceptions.

Now I did go down the route of half implementing with Threading.Timer

  1. Don't use System.Windows.Forms.Timer because it won't work (this only makes sense).

  2. Don't use System.Threading.Timer because it doesn't work, use System.Timers.Timer instead.

  3. Don't use System.Timers.Timer because it doesn't work, use System.Threading.Timer instead.

Related question.

I really could not be bothered with the issues that came with these steps, as my aim was to focus on the business logic. So I made the decision to use Quartz.Net.

It made my life much easier and seems to work perfectly! 45-60 minutes worth of work.

Basic Setup:

1, Nuget > Quartz

2, New Folder(s) > QuartzComponents > Jobs and Schedule

enter image description here

3, Added OrderJob.cs and OrderJobSchedule.cs in respective folders

OrderJob.cs logic:

 public sealed class OrderJob : IJob
 {
    public void Execute(IJobExecutionContext context)
    {
            var orderUoW = new OrderUoW();
            orderUoW.Create();
     }
  }

Notice that it creates and instance of my UoW?!? logic that it would hit on each pass.

OrderJobSchedule.cs logic:

   public sealed class OrderJobSchedule
    {
        public void Start()
        {
            IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
            scheduler.Start();

            IJobDetail job = JobBuilder.Create<OrderJob>().Build();

            ITrigger trigger = TriggerBuilder.Create()
                   .WithSimpleSchedule(a => a.WithIntervalInSeconds(15).RepeatForever())
                   .Build();

            scheduler.ScheduleJob(job, trigger);
        }

        public void Stop()
        {
            IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
            scheduler.Shutdown();
        }
    }

A lot of magic here, but to emphasis on:

JobBuilder.Create<OrderJob>().Build();

a.WithIntervalInSeconds(15).RepeatForever()

Now that's in place we need to add logic to the "guts" of the service:

public partial class QuayService : ServiceBase
    {
        OrderJobSchedule scheduler;
        public QuayService()
        {
            InitializeComponent();
        }
        protected override void OnStart(string[] args)
        {
            try
            {
                scheduler = new OrderJobSchedule();
                scheduler.Start();

                SendEmail("(Shopify)Quay Service Started",
                    "(Shopify)Quay Service Successfuly Started");
            }
            catch (Exception ex)
            {
                ProcessException(ex, "Error starting (Shopify)Quay Service");
                EventLog.WriteEntry("Error starting (Shopify)Quay service and timer..." + ex.Message);
            }
        }
        protected override void OnStop()
        {
            try
            {
                if (scheduler != null)
                {
                    scheduler.Stop();
                }

                SendEmail("(Shopify)Quay Service stopped",
                    "(Shopify)Quay Service Successfuly Stopped");
            }
            catch (Exception ex)
            {
                ProcessException(ex, "Error stopping (Shopify)Quay Service");
                EventLog.WriteEntry("Error stopping (Shopify)Quay timer and service..." + ex.Message);
            }
        }
        private void SendEmail(string subject, string body)
        {
            new Email().SendErrorEmail("Quay", subject, body);
        }
        private void ProcessException(Exception ex,
            string customMessage)
        {
            var innerException = "";
            if (ex.InnerException != null)
                innerException = (!string.IsNullOrWhiteSpace(ex.InnerException.Message)) ? ex.InnerException.Message : "";


            new Email().SendErrorEmail("Quay", customMessage, 
                ex.Message + " " + innerException);
        }
    } 

Very easy to set up and solved my horrendous experience with Timers.Timer Whilst I didn't fix the core issue, I came up with a solution and got a bug free working system in place.

Please note to future readers DO NOT USE System.Timers.Timer unless you are prepared to add a "hack'ish" fix.

Community
  • 1
  • 1
Tez Wingfield
  • 2,129
  • 5
  • 26
  • 46
  • It's a good rewrite. I think the points 2 & 3 are conflicting in your answer: **2, Don't use System.Threading.Timer because it doesn't work, use System.Timers.Timer instead.** **3, Don't use System.Timers.Timer because it doesn't work, use System.Threading.Timer instead.** – Imran Saeed Jan 17 '17 at 11:19
  • @i.net Thanks... I pulled that from another SO thread as it was the feeling I was getting. I think the point being, is they are supposed to be conflicting. They both are a pain in the "beep" to get working and at least one of them is filled with bugs, – Tez Wingfield Jan 17 '17 at 12:59