setjmp and longjump to implement threads

Question

I had a question about using setjmp and longjump to create function stacks that can run independently of one another. With reference to this question

Here the function stack for B() seems to be on top of the one for A so when A goes out of scope and I try longjumping to B() the code segfaults. The modified code looks like so

#include <stdio.h>
#include <setjmp.h>
#include <iostream>
using namespace std;

jmp_buf bufferA, bufferB;

void routineB(); // forward declaration 

void routineA()
{
    int r ;

    printf("(A1)\n");

    r = setjmp(bufferA);
    if (r == 0) routineB();

    printf("(A2) r=%d\n",r);

    r = setjmp(bufferA);
    if (r == 0) longjmp(bufferB, 20001);

    printf("(A3) r=%d\n",r);

    r = setjmp(bufferA);
    if (r == 0) longjmp(bufferB, 20002);

    printf("(A4) r=%d\n",r);
}

void routineB()
{
    int r;

    printf("(B1)\n");

    r = setjmp(bufferB);
    if (r == 0) longjmp(bufferA, 10001);

    printf("(B2) r=%d\n", r);

    r = setjmp(bufferB);
    if (r == 0) longjmp(bufferA, 10002);

    printf("(B3) r=%d\n", r);

    r = setjmp(bufferB);
    if (r == 0) longjmp(bufferA, 10003);

    cout << "WHAT" << endl;
}


int main(int argc, char **argv) 
{
    routineA();
    longjmp(bufferB, 123123);
    return 0;
}

I was thinking, we can have one main function each coroutine (in this case B and A) jumps back to which then in turn jumps to one of the coroutines given that they are alive. Will that work or will that also segfault because some of the stacks for coroutines that are possibly dead are on top of ones that want to run?

If it does seem like it should segfault, then how can one implement such independent couroutines (which can run when others have died and don't depend on each other) in C++?

NOTE: I do not want to use swapcontext, makecontext, setcontext, and getcontext since they are deprecated. The purpose of this post is to help me find alternatives to those functions

Thanks in advance!

Answer 1

It should work, with some precautions. It used to work in the nineties.

It works as long as the thread that is older in the stack does not return, then there should be no issue. For a generator or a thread, it means that it terminates with some final yield, but no return.

I remember asking IBM to fix their AIX implementation of setjmp/longjmp so that it would stop checking (and forbidding) that usefull behaviour that other implementation allowed. That was the most portable way I found back then to implement cooperative threads.

You can do even better with a state machine based stack allocator in order to allocate/free stacks of different sizes to different thread.

That's very tricky however as far I a remember. I remember Matz (from Ruby) asking me years later about the code for that but alas I had no more access to it at the time.

Anyways, this does not matter much nowadays because pthread does the job.

Answer 2

Quick and dirty hack on Windows. Sorry for the messy naming and messy code.

    // mythreads.cpp : Defines the entry point for the console application.
//

#include <setjmp.h>
#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include "List.h"

#include <Windows.h>

struct thread_t 
{
  util::Node node;
  jmp_buf buf;
};

util::List thread_list;
thread_t* runningThread = NULL;

void schedule(void);

void thread_yield()
{
  if (setjmp(runningThread->buf) == 0)
  {
    thread_list.push_back(&runningThread->node);
    schedule();
  }
  else
  {
    /* EMPTY */
  }
}

void myThread1(void *args)
{
  printf("myThread1\n");
}

void startThread(void(*func)(void*), void *args)
{
  thread_t* newThread = (thread_t*)malloc(sizeof(thread_t));

  // Create new stack here! This part is Windows specific. Find the current stack
  // and copy the contents. One might do more stuff here, e.g. change the return address 
  // of this function to be a thread_exit function.
  NT_TIB* tib = (NT_TIB*)__readfsdword(0x18);
  uint8_t* stackBottom = (uint8_t*)tib->StackLimit;
  uint8_t* stackTop = (uint8_t*)tib->StackBase;

  size_t stack_size = stackTop - stackBottom;
  uint8_t *new_stack = (uint8_t*)malloc(stackTop - stackBottom);
  memcpy(new_stack, stackBottom, stack_size);

  _JUMP_BUFFER *jp_buf = (_JUMP_BUFFER*)newThread->buf;

  if (setjmp(newThread->buf) == 0)
  {
    // Modify necessary registers to point to new stack. I may have 
    // forgotten a bunch of registers here, you must do your own homework on
    // which registers to copy.
    size_t sp_offset = jp_buf->Esp - (unsigned long)stackBottom;
    jp_buf->Esp = (size_t)new_stack + sp_offset;
    size_t si_offset = jp_buf->Esi - (unsigned long)stackBottom;
    jp_buf->Esi = (size_t)new_stack + si_offset;
    size_t bp_offset = jp_buf->Ebp - (unsigned long)stackBottom;
    jp_buf->Ebp = (size_t)new_stack + bp_offset;
    thread_list.push_back(&newThread->node);
  }
  else
  {
    /* This is where the new thread will start to execute */
    func(args);
  }
}

void schedule(void)
{
  if (runningThread != NULL)
  {

  }

  if (thread_list.size() > 0)
  {
    thread_t* t = (thread_t*)thread_list.pop_front();
    runningThread = t;
    longjmp(t->buf, 1);
  }
}

void initThreading()
{
  thread_list.init();

  thread_t* mainThread = (thread_t*)malloc(sizeof(thread_t));

  if (setjmp(mainThread->buf) == 0)
  {
    thread_list.push_back(&mainThread->node);
    schedule();
  }
  else
  {
    /* This is where the main thread will start to execute again */
    printf("Main thread running!\n");
  }
}

int main()
{
  initThreading();

  startThread(myThread1, NULL);

  thread_yield();

  printf("Main thread exiting!\n");
}

Also, Microsoft and setjmp/longjmp might not be a good match. If I have had a Unix box nearby I would have done it on that one instead.

And I would probably have to study a bit more about the stack copy to make sure it works correctly.

EDIT : To check which registers to modify after stack change one might consult the compiler manual.

The code starts in main() by setting up the threading framework. It does this by treating your main thread (the only thread at this point) as just a thread among others. So, store the context for the main thread and put it in the thread_list. Then we schedule() to let one thread run. This is done in initThreading(). Since the only thread running is the main thread, we will continue in the main() function.

The next thing the main function does, is startThread with a function pointer as parameter (and the arg NULL to be sent to the func). What the startThread() function does is that it allocates memory for a stack for the new thread (every thread needs its own stack). After allocation, we save the context of the running thread into the new thread context buffer (jmp_buf) and change the stack pointers (and allo other registers that should be changed) so that it points to the newly created stack. Then we add this new thread to the list of waiting threads. And then we return from the function. We are still in the main thread.

In the main thread we do thread_yield() which says "OK, I don't want to run anymore, let someone else run!". The thread_yield function stores the current context and puts the main thread at the back of the thread_list. Then we schedule.

Schedule() takes the next thread from thread_list and does a longjmp to the context saved in the thread buf (jmp_buf). In this case the thread will continue to run in the else clause of the setjmp where we stored the context.

  else
    {
        /* This is where the new thread will start to execute */
        func(args);
      }

It will run until we do a thread_yield and then we will do the same thing, but instead take the main thread and do a longjmp on its saved buffer and so on and so on...

If one wants to be fancy and want time slices, one can implement some alarm to save the current thread context and then call schedule().

Any clearer?