3

For the past couple of days I've been struggling with a really strange issue. I have a .net-core 2.1 console app that has been thoroughly tested on Windows 10 and works perfectly there. However, when deploying it for Ubuntu 18.04 and running it there, the app starts behaving strangely. I deploy it like so:

dotnet publish .\MyProject.csproj -c Release -r ubuntu.18.04-x64 --output ./published

What the console app does essentially, is run a function every second, which runs multiple threads using the ThreadPool class (up to 4). Each of those threads from the threadpool do some calculations and write the results in a database. The threads are independent from each other, don't communicate, don't wait for others to finish, and use separate instances of the same class to do the calculations and db writing. This is how the threads are invoked each second:

foreach (var item in items)
{
    ThreadPool.QueueUserWorkItem(obj =>
    {
         DoTheCalcAndWriteToDb();
    });
}

The expected output, which works on Windows, is each of the threads printing their results to the console and printing the second for which the function was run for. Here is the log example for second 45 and 46:

Thread 1 info: Function done in second: 45:464
Thread 2 info: Function done in second: 45:464
Thread 3 info: Function done in second: 45:484
Thread 4 info: Function done in second: 45:504
---
Thread 1 info: Function done in second: 46:445
Thread 2 info: Function done in second: 46:445
Thread 3 info: Function done in second: 46:454
Thread 4 info: Function done in second: 46:485

As you can see, each thread is invoked at each second through the ThreadPool, and finished at the same second.

Now the behavior on Linux (Ubuntu) is completely different. Running it for a few seconds gives the same results and seems to work as same as on Windows, but after a few minutes (never the same time) the logs become inconsistent. Some print after a few seconds, and others don't print at all. After the problems start they never return to the regular state. Of course, it's not just the printing, the db also doesn't have new data inserted in it every second (it should have 4 new fields in it, every second).

I'm sorry for the limited information, but I currently can't determine where the problem is or what other info to give you in order to figure out what is causing the issue.

Does anyone have any idea where the problem might be? I can provide more info regarding the calculations and the db work, but it's really nothing special just some math and then write that to a new field in RethinkDB.

Marko Grbic
  • 91
  • 1
  • 1
  • 7
  • To a first approximation, if you have only up to 4 threads, there is no real point in using the thread pool. You may as well have 4 *dedicated* threads (or tasks) to perform the work. A big potential problem with this approach is that `QueueUserWorkItem` has no native way of communicating completion of anything; this means that if a thread, for whatever reason, ceases to do useful work, or you issue more requests than can complete, it's basically impossible to see this (as the pool will happily spin up more threads to compensate, until it reaches the maximum and things stall). – Jeroen Mostert Jun 28 '19 at 12:11
  • I was using a ThreadPool for faster thread creation and initialization. Time is very critical in the application, and a 20-30ms start up time for creating a new Thread each second is a lot.. – Marko Grbic Jun 28 '19 at 12:55
  • I'm not suggesting creating new threads *each second*! That would be absurd. Creating four threads at the beginning, however (or four long-running `Tasks`) should not be markedly slower than waiting for the thread pool to create them -- the thread pool does not, as far as I know, have access to magic thread creation functions that are faster than what code can explicitly manage. If you're not doing any more than performing an action/actions each second, you could also just use a `System.Threading.Timer` (or four). In general, a `Task` is also more manageable than `QueueUserWorkItem`. – Jeroen Mostert Jun 28 '19 at 13:07
  • Creating 4 or N threads at the beginning was the first idea I had, but I didn't find a way to reuse a thread once it has finished it's work. Only way I could reuse existing threads is never let them finish in the first place and just send out signals for them to run some actions again? – Marko Grbic Jun 28 '19 at 13:16
  • Also, since milliseconds are important in the application, using a timer didn't do well with it's ~15ms precision. – Marko Grbic Jun 28 '19 at 13:17
  • The idea would be to keep the threads alive, yes. This makes more sense if they don't actually receive any data, but work on their own, since communicating with threads is clumsy. (You can use things like `BlockingCollection` to make queues, though.) If millisecond precision matters, `QueueUserWorkItem`/arbitrary tasks is not what you want, because (as the name implies) all it does is queue work to be taken up when it's convenient. Counting on this to happen in a particular time frame is asking for trouble. – Jeroen Mostert Jun 28 '19 at 13:24
  • Okay, makes sense. I'll try that. Thanks. – Marko Grbic Jun 28 '19 at 13:26
  • 1
    With regards to timer precision: if you just need things to happen "on the dot" (i.e. the problem is just the timer drifting), the 15 ms system timer precision is not generally a major problem: you can just adjust the time to the next interval based on the clock (see [this question](https://stackoverflow.com/q/9228313/4137916), for example). I can't imagine you have hard real-time requirements if you have to interface with an external system; if you do, things get much more complicated anyway (as neither Windows nor Linux are really geared towards that). – Jeroen Mostert Jun 28 '19 at 13:27

0 Answers0