Implementing a profiler for worker thread tasks

Question

I have a number of worker threads which work on tasks called class Task. I'm using C++ on x86_64 Windows/Mac/Linux. While working on each task, I can update a global Task* activeTasks[] array so that the application knows when each worker is processing which task.

activeTasks[threadId] = &task;
task.work();
activeTasks[threadId] = NULL;

I would like to write a simple in-application profiler so that the user can see how many microseconds have been spent on each task.

An additional complication is that tasks may call sleep() inside their work() function. The profiler should only sample a task when the thread is active, not sleeping or suspended by the scheduler.

How could a profiler like this be implemented? It seems this is exactly how profilers like perf work, except they inspect the current call stack rather than an activeTasks array.

Attempts

A naive idea is to launch a separate profiler thread that periodically checks activeTasks[threadId] every few microseconds, incrementing a counter for each task if the worker thread is in a running state. This can be checked with thread.ExecutionState == 3 on Windows and possibly some way with pthreads. But the issue is that on single-core machines, running a profiler thread and any of the worker threads is never simultaneous, so the profiler would always see the worker as "suspended".

Another idea is to trigger some sort of interrupt, but I have no idea how this is done.

You cannot profile something outside your control, i.e. the scheduler. You have to rewrite the Sleep() function to trigger off your profiler, then on timeout trigger it on again. Interrupts cannot be triggered in a user level application. — Michael Chourdakis, Aug 09 '19 at 16:27
Outside of what @MichaelChourdakis pointed out, you cand simply wrap your worker functions in a _timer_ function as I explain [here](https://stackoverflow.com/a/22387757/1673776). Also, you can update your `activeTasks` via a mutex so you prevent race conditions. — Victor, Aug 09 '19 at 16:29
I hope you are protecting any access (read or write) to `activeTasks` with a mutex or other synchronization primitive. Otherwise you have a data race and are deep into undefined behaviour land (and you *do not* want to be there). — Jesper Juhl, Aug 09 '19 at 16:30
@Victor Of course, but I didn't think the thread-safety of my pseudocode was relevant to this question. — Vortico, Aug 09 '19 at 16:32
@Victor But would that timer function incorrect count time spent sleep()ing? — Vortico, Aug 09 '19 at 16:32

score 2 · Accepted Answer · answered Aug 09 '19 at 16:38

2

It sounds like you want to measure thread CPU time.

There are ways to do that in Win32 (using GetThreadTimes) and pthreads (using pthread_getcpuclockid).

long startTime = getThreadCPUTime();
task.work();
long endTime = getThreadCPUTime();
// lock mutex
taskTimes[taskType] += endTime - startTime;
// unlock mutex

answered Aug 09 '19 at 16:38

rustyx

80,671
25
200
267

Thanks, your references to the two "thread CPU time" functions are the key. I didn't know this existed for both platforms, and it solves the issue of inactive threads without periodic sample profiling. – Vortico Aug 09 '19 at 17:20

Michael Chourdakis · Answer 2 · 2019-08-09T16:40:13.480

1

The simple implementation is to have a class that can be toggled on/off, then at the end of the (say) function log a counter. Having another thread doing that is bad because you would consume CPU time on polling time, which is bad.

class PROFILE
{
 private: unsigned long long start = 0,consumedtime = 0;
 public:
    PROFILE()
    {
     on();
    }
    ~PROFILE()
    {
     off();
     log(consumedtime); // e.g. save to a global array with a mutex
    }

    void on()
    {
     start = some_tick(); // E.g. GetTickCount64() on Windows
    }
    void off()
    {
     auto end = some_tick();
     consumedtime += end - start;
    }

   void sleep(int ms)
   {
    off();
    Sleep(ms); // Win
    on();
   }
}

void foo()
{
    PROFILE pr;
    ...
    pr.sleep(500);
    ...
}

edited Aug 09 '19 at 16:40

answered Aug 09 '19 at 16:30

Michael Chourdakis

10,345
3
42
78

Is the difference between `some_tick()` and `std::chrono::high_resolution_clock::now()` that the former is thread-time and while latter is in real-time? In other words, if `work()` has `sleep()`, will the former not count the time sleeping? – Vortico Aug 09 '19 at 16:34
some_tick() is any function that can give you a current tick, I edited to provide an off/on function. If you sleep during a function, you must trigger the profiler off manually. – Michael Chourdakis Aug 09 '19 at 16:36
Thanks, if combining this implementation with @rustyx's answer, I believe is a sufficient alternative to sample profiling. – Vortico Aug 09 '19 at 17:22

Implementing a profiler for worker thread tasks

Attempts

2 Answers2