0

I would like to enable/set the trap flag on an x86_64 machine, then execute some instructions, then disable/unset the trap flag again. When trapping, I want my own handler to be called. My attempt to do this is below.

It does not work as expected, instead, the trap seems to happen and my program is aborted. Why is that so and how can I fix it?

The output I currently see is this (on a linux 5.10.0 machine):

./test 
Starting execution...
Trace/breakpoint trap
#include <assert.h>
#include <signal.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>

void setTrapFlag() {
  asm volatile("pushfq\n" // push status register to stack
               "xorl $0x100, (%rsp)\n" // set trap-flag of on-stack value 
               "popfq\n" // pop status register
               );
}

void trapHandler(int signo, siginfo_t *info, void *context) {
  setTrapFlag();
}


int main ()
{
 struct sigaction trapSa;
  
 // set up trap signal handler
  trapSa.sa_flags = SA_SIGINFO;
  trapSa.sa_sigaction = trapHandler;
  int ret = sigaction(SIGTRAP, &trapSa, NULL);
  assert(ret == 0);

 printf("Starting execution...\n");
 setTrapFlag();
 printf("Set trap flag!\n");
 setTrapFlag();
 printf("UnSet trap flag!\n");

 printf ("all done\n");
 return 0;
}

I was expecting an output of

./test
Starting execution...
Set trap flag!
UnSet trap flag!
all done
  • I haven't ever used this before, but isn't the Trap Flag what amounts to a single-step mode? I don't know how a program can single-step itself without outside (debugger) help...? – Steve Friedl Jan 06 '23 at 20:56
  • Exactly! The idea of this is to execute a single instruction, log something about the address in the trap signal handler, then execute the program normally - note that this is a minimal example about the part of the code that doesn't work as expected. It is not a complete example that does something useful. – tim0s42 Jan 06 '23 at 21:07
  • It is possible to do this. But, the easy way would be to use the existing `ptrace` syscalls. That's what they're designed for. They are what `gdb`, `strace`, etc. use to single step programs. – Craig Estey Jan 06 '23 at 21:17
  • I don't believe the trap flag means "single step the next instruction", it means "turn on single-step until turned off". Aren't the trap handler's instructions consider part of the next instruction stream? – Steve Friedl Jan 06 '23 at 21:17
  • @CraigEstey can a program ptrace (or at least single-step via ptrace) *itself*? – Steve Friedl Jan 06 '23 at 21:24
  • 1
    @SteveFriedl A "program" can. It can `fork` itself. The child does `PTRACE_TRACEME` and then the parent can trace/step/control the child. It's similar to what `gdb` would do, except the program already has addresses of functions, etc (i.e. the symbol table). to be able to breakpoint things, etc. – Craig Estey Jan 06 '23 at 21:41
  • I got the idea from here: http://ant6n.ca/2017-01-11-trapflag-tracing/ so apparently a program can trace itself, and it is much faster than forking off a copy first. – tim0s42 Jan 06 '23 at 21:44
  • What I am actually trying to achieve is also outlined here: https://stackoverflow.com/questions/21068714/trap-all-accesses-to-an-address-range-linux --- so it seems to be that using traps is the right approach, the question is why it doesn't work for me :) – tim0s42 Jan 06 '23 at 21:54
  • On a modern OS, a `fork` takes little overhead because it does _not_ copy the address space. It just creates an MMU mapping. Note that in your link to ant6n.ca, it does a `fork`. If one uses `ptrace` to _self_ trace [if it's even possible], I think that most of the "debugger" will have to operate from within the signal handler. So, it is limited to what calls are "signal safe". So, no `malloc` et. al. – Craig Estey Jan 06 '23 at 21:54
  • Your SO link is, more or less, what I was calling "self trace". While setting/clearing TF in a kernel or bare metal environment works great, it is the somewhat "bulky" nature of a signal handler (vs. a driver/kernel interrupt service routine) that makes this more difficult. Having written ROM based debuggers before, I'd say, under linux, just use `ptrace` as it was intended (a separate parent process). From the perspective of the traced child, the parent process is similar to a "kernel". The parent can't be borked by the child – Craig Estey Jan 06 '23 at 22:02

0 Answers0