3

I currently have an application which is basically a wrapper for ~10 "LongRunning" Tasks. Each thread should keep running indefinitely, but sometimes they lock up or crash, and sometimes the wrapper app spontaneously exits (I haven't been able to track that down yet). Additionally, the wrapper application can currently only be running for one user, and that user has to be the one to restart the threads or relaunch the whole app.

I currently have a monitor utility to let me know when the threads stop doing work so that they can be manually restarted, but I'd like to automatically restart them instead. I'd also like the wrapper to be available to everyone to check the status of the threads, and for the threads to be running even when the wrapper isn't.

Based on these goals, I think I want to separate the threads into a Windows Service, and convert the wrapper into something which can just connect to the service to check its status and manipulate it.

How would I go about doing this? Is this a reasonable architecture? Should I turn each thread into a separate service, or should I have a single multi-threaded service?

Edit: All the tasks log to the same set of output files (via a TextWriter.Synchronized(StreamWriter)), and I would want to maintain that behavior.
They also all currently share the same database connection, which means I need to get them all to agree to close the connection at the same time when it's necessary. However, if they were split up they could each use their own database connection, and I wouldn't need to worry about synchronizing that. I actually suspect that this step is one of the current failure points, so splitting it up would be a Good Thing.

Bobson
  • 13,498
  • 5
  • 55
  • 80

2 Answers2

2

I would suggest you to stay inside one multithreading service if possible. Just make sure that threads are handled correctly when Service Stop is triggered. Put brake flags inside blocks of code that will take a lot of time to execute. This way you will make your service responsive on Stop event. Log any exceptions and make sure to wait for all threads to exit until service is finally stopped. This will prevent you to run same "task" in multiple threads.

Maintaining one service is in the end easier then multiple services.

Splitting to multiple services would be reasonable if you require some separate functionalities that can run or not beside each other.

Gregor Primar
  • 6,759
  • 2
  • 33
  • 46
  • With this model, how would I handle the lockup scenario where I want to restart one unresponsive thread out of the set? – Bobson Nov 27 '12 at 15:13
  • From complete service perspective it's not possible. Restarting should restart all threads. However you can declare each thread as private inside main service thread and monitor them. If you detect spceific thread as unresponsive just abort it and start new thread or you can decide to restart complete service. – Gregor Primar Nov 27 '12 at 15:22
  • Unfortunately, due to other architectural compromises, every thread in the same application uses the same database connection, and if that connection goes unresponsive then there's no way to check to see if the threads are doing work. Hence the monitor utility, which has its own connection. Would it be possible to send a message to the service to tell it "Restart thread X"? – Bobson Nov 27 '12 at 15:32
  • You can check if thread is alive http://msdn.microsoft.com/en-us/library/system.threading.thread.isalive.aspx . Alternative would be to record DataTime when specific thread has done some work (like connecting to DB). About restarting from outside of the service. Yes this is possible I have done similar thing but it will require additional work. You would have to writte tcp/ip or wcf server inside your service and recieve commands from clients (external app). Of course then also restarting specific thread on demand would be possible. – Gregor Primar Nov 27 '12 at 15:40
  • 1
    You inspired me to go find [this](http://stackoverflow.com/a/3695266/298754), which I think should address the restarting a thread issue. Combining that with your suggestion should make a perfectly workable solution. – Bobson Nov 27 '12 at 16:07
  • Yes this will work for restarting issue, however you will not be able to return any service state (hanging connection), since ExecuteCommand is void with no our or ref parameters. – Gregor Primar Nov 27 '12 at 17:39
0

I don't think moving the threads to a Windows Service removes any of the problems. The service will still crash randomly and the threads will still exit randomly.

I assume that your long-running tasks implement a kind of worker loop. Wrap the body of that loop in a try-catch and log all exceptions. Don't rethrow them so that the task does not ever exit. Examine the logs to find the bugs.

usr
  • 168,620
  • 35
  • 240
  • 369
  • It wouldn't fix any of the underlying problems, no. However, it would make it feasible to automatically kill and restart them, and separate the control wrapper's UI from the workers. – Bobson Nov 27 '12 at 15:09
  • My main point is that moving to a service does not help. What do you think about that? – usr Nov 27 '12 at 15:11
  • Unless there's a way to automatically kill and restart threads in one application from another app, then moving it to a service would help. I know how to kill and restart a service. Obviously, it would be *better* to find and fix the bugs, but I can't ever guarantee that the thread hasn't just locked up while waiting for something (which is a recurring issue with our database). The only way I can tell *that* is to see whether work is being done. – Bobson Nov 27 '12 at 15:21
  • Doesn't killing a service cause service interruption? It seems to me that you probably want to restart a single thread only if it dies or becomes unresponsive. Or better yet, have a process per thread and restart the process. That is probably most reliable.; Anyway, you don't need a windows service. You can just use a child process and keep a handle to it to detect its death. – usr Nov 27 '12 at 15:37
  • Killing it and bringing it back up shortly thereafter wouldn't have any repercussions. But just restarting the one thread would be ideal. The problem is that the app itself can't detect when a thread has hung due to the database connection being lost, because it needs that connection to tell whether the thread has hung. See my comment on Gregor's answer. – Bobson Nov 27 '12 at 16:03
  • Again, I don't think having service adds anything. You can implement the same kind of messaging in-process or with a child process. Executing in a service complicates things. It requires installation and such. It makes debugging harder. I don't recommend it (from experience). – usr Nov 27 '12 at 16:32