4

We see a couple of below mentioned messages in /var/log/messages for one of our application:

Sep 18 03:24:23 <machine_name> kernel: application_name[14682] trap invalid opcode rip:f6c6e3ce rsp:ffc366bc error:0
...
Sep 18 03:19:35 <machine_name> kernel: application_name[4434] general protection rip:f6cd43a2 rsp:ffdfab0c error:7b2

I am not able to make what’s these output means and how we can track the function / code that is causing the issue. Further what is 'trap invalid opcode' and 'general protection' means?

jww
  • 97,681
  • 90
  • 411
  • 885
Prakash
  • 41
  • 1
  • 1
  • 3

2 Answers2

2

Usually that means that your program's instruction pointer points to data or garbage. That's commonly caused by writing to stray pointers and such.

One scenario would be that your code writes (through a stray pointer) over some class' virtual table, replacing the member function addresses with nonsense. The next time you call one of the class' virtual functions, your program will interpret the garbage as an address and jump to that address. If whatever data lies at this address happens to not to be a valid machine code instruction for your processor, you would see this error.

sbi
  • 219,715
  • 46
  • 258
  • 445
2

There is another possibility that can cause 'invalid' op codes, that would be hardware not supporting newer opcode/instruction sets(SSE 4/5) or it not being from the right manufacturer(both AMD and Intel have some specific opcodes that work only on their processors) or just not having permission to exectute certain ops(though this would probably show up as something else).

From the above I would take RIP to be 'register(?) instruction pointer' and RSP to be 'register stack pointer', in which case you could use a debugger and set an execution hardware breakpoint on the specified address(RIP) and trace back what is calling it.(it seems your using linux or unix, so this is quite vague). if you are on windows, try using a custom exception filter to capture the EXCEPTION_ILLEGAL_INSTRUCTION event to get a little more information

Necrolis
  • 25,836
  • 3
  • 63
  • 101
  • We are using Linux and is there anyway using a tool that we can identify where the issue is? What does RIP and RSP values actually means - are they function address. If so looking for address greater than that will hint the location appx which is giving the issue. But who do you generate a mapfile of an executable both for 64 and 32 bit architecture. – Prakash Sep 21 '10 at 09:33
  • unfortunatly my linux knowledge is very sparse(I'm a windows only person, out of shear lazyness), but yes, RIP and RSP should be memory pointers, if the pointers are the same all or most of the time, you would breakpoint at that location(RIP), else you need to let the debugger trap the invalid events and backtrace the call stack. – Necrolis Sep 21 '10 at 09:59