Expanding on a Job/Worker multi-threading system in C++

Question

I've been reading a tutorial by Ben Hoffman (https://benhoffman.tech/cpp/general/2018/11/13/cpp-job-system.html)

I've had a go at bashing together a version of the Job/Worker system he has, but instead of using void* for arguments then casting to a known struct, I've been trying to use variadic arguments. The idea is, a job takes in a "parent" to perform a method on, the function pointer to said method, and an Args... for the argument(s). However, I get an internal compiler error if I try to build. Here is the job class:

template <class T, typename... Args>
struct JobMemberFunc : IJob
{
    JobMemberFunc(T* aParent, void (T::* f)(Args...), Args... Args)
    {
        parentObj = aParent;
        func_ptr = f;
        saved_args = ::std::make_tuple (::std::move(Args)...);
    }

    virtual bool invoke() override
    {
        if (!parentObj) { return false; }

        (parentObj->*func_ptr)(::std::move(saved_args));
        return true;
    }

    /** the object to invoke the function pointer on */
    T* parentObj;

    /** The function pointer to call when we invoke this function */
    void (T::* func_ptr)(Args...);

    ::std::tuple<Args...> saved_args;
};


struct CpuJob
{
    IJob* jobPtr = nullptr;
};

Then there's the AddJob method, where the internal compiler error is actually happening.

template <typename T, typename... Args>
void AddJob(T* aParent, void(T::* func_ptr)(Args...), Args... args)
{//This curly bracket is where the internal compiler error happens
    CpuJob aJob = {};

    JobMemberFunc<T, Args...>* jobPtr = new JobMemberFunc<T, Args...>(aParent, func_ptr, 
    std::forward<Args>(args)...);

    aJob.jobPtr = jobPtr;

    locklessReadyQueue.enqueue(aJob);
}

More than happy to be told this is a bad/wrong way of trying to do it anyway. I have thought about doing away with it and having a standardized argument list or doing something polymorphic but I really wanna make this work so I can literally ask the job system to do anything I like.

Thanks!

Personally I think a job is just a `void(void)` and then anything more complex than that is a lambda. Keeping it simple is vastly underrated. — UKMonkey, Oct 18 '19 at 11:33
It's likely not the error but the line `(parentObj->*func_ptr)(::std::move(saved_args));` will cause trouble since `func_ptr` expects multiple arguments but you've put them all into a tuple. You will have to unwrap them before call the member function. — Albjenow, Oct 18 '19 at 11:41
Because I'm trying to write a game engine, and spawning threads all over the place is generally a bad idea. I'm hoping to centralize it all, and then when I want to do the AI calculations or some matrix maths I can send it to the job centre and it's done on another thread without locking the main one — Figwig, Oct 18 '19 at 11:42
@Albjenow doesn't the `::std::move()` unwrap them for me? What do I need to do to unwrap them? — Figwig, Oct 18 '19 at 11:43
Will Hain: Both do not mean to spawn threads all over the place, on the contrary they are designed for uses cases where you want to control the threading. — Superlokkus, Oct 18 '19 at 11:46
@Superlokkus right okay I'll have to look into them more. Still feels weird to not have it centralised, RDD n that ringing alarm bells but will look at it — Figwig, Oct 18 '19 at 11:54

score 2 · Accepted Answer · answered Oct 18 '19 at 12:13

2

std::function<void()> (in combination with lambdas) already do what you're trying to do with JobMemberFunc.

void AddJob(std::function<void()>&& job)
{
    locklessReadyQueue.enqueue(std::move(job));
}

With this you can submit any function call as a job.

For example, a call some_obj.some_method(some_arg) becomes:

  AddJob([&] { some_obj.some_method(some_arg); });

No more ugly pointer-to-member stuff...

You can find more complete thread pooling examples here: Thread pooling in C++11

answered Oct 18 '19 at 12:13

rustyx

80,671
25
200
267

That looks much nicer. I'll have a go with that. Thank you – Figwig Oct 18 '19 at 12:14
Not only does it work, it's taken all the template stuff out and makes it much more readable. Thank you for your simple solution – Figwig Oct 18 '19 at 12:43

Expanding on a Job/Worker multi-threading system in C++

1 Answers1