Why does my kernel crash when I add more arguments to a function?

Question

I am trying to build a custom kernel in C and I want to put a colored pixel on the screen.
I already switched into protected mode and set the VGA mode to 0x13, so I wrote a function in C to put a pink pixel on the right upper corner of the screen:

#include <stdint.h>

#define VGA_MODE 0x13
#define VRAM 0xA0000
#define MAX_HEIGHT 200
#define MAX_LENGTH 320

void PutPixel(uint8_t color, uint16_t x)
{
    // Return if x greater than 320
    if(x > MAX_LENGTH) return;

    unsigned char* ptr = (unsigned char*) VRAM;

    ptr += x;

    *ptr = color;

    return;
}

void KERNEL_MAIN()
{
    PutPixel(5, 320);

    return;
}

This code runs perfectly fine, until I want to add a y-offset as parameter to my PutPixel function in order to draw a pixel on another row:

#include <stdint.h>

#define VGA_MODE 0x13
#define VRAM 0xA0000
#define MAX_HEIGHT 200
#define MAX_LENGTH 320

void PutPixel(uint8_t color, uint16_t x, uint16_t y)
{
    // Return if x greater than 320
    if(x > MAX_LENGTH) return;

    unsigned char* ptr = (unsigned char*) VRAM;

    ptr += x;

    *ptr = color;

    return;
}

void KERNEL_MAIN()
{
    PutPixel(5, 320, 1);

    return;
}

Even when I don't actually use the parameter, when I run the code in VirtualBox, it crashes immediately and I get following error in the VirtualBox debugger:

dbgf event: DBGFSTOP (hyper)
File:     VINF_EM_TRIPLE_FAULT
Line:     0
Function: <NULL>
eax=000a013e ebx=00007eae ecx=00000005 edx=00000005 esi=00000000 edi=0000fff0
eip=00000083 esp=0002ffac ebp=490002ff iopl=0 nv up di pl nz na pe nc
cs=0008 ds=0010 es=0010 fs=0010 gs=0010 ss=0010               eflags=00200002
0008:00000083 f0 53                   Illegal opcode
VBoxDbg>

Can I get some help please?

EDIT: I found that the stack got messed up by one byte, so I disassembled the function call and it looks like this:

%0000000000007ee7 55                      push ebp
%0000000000007ee8 48                      dec eax
%0000000000007ee9 89 e5                   mov ebp, esp
%0000000000007eeb 48                      dec eax
%0000000000007eec 83 ec 10                sub esp, byte 000000010h
%0000000000007eef 89 d0                   mov eax, edx
%0000000000007ef1 44                      inc esp               ;???
%0000000000007ef2 89 c2                   mov edx, eax
%0000000000007ef4 88 4d 10                mov byte [ebp+010h], cl
%0000000000007ef7 66 89 45 18             mov word [ebp+018h], ax
%0000000000007efb 89 d0                   mov eax, edx
%0000000000007efd 66 89 45 20             mov word [ebp+020h], ax
%0000000000007f01 66 81 7d 18 40 01       cmp word [ebp+018h], word 00140h
%0000000000007f07 77 1c                   jnbe +01ch (000007f25h)
%0000000000007f09 48                      dec eax
%0000000000007f0a c7 45 f8 00 00 0a 00    mov dword [ebp-008h], 0000a0000h
%0000000000007f11 0f b7 45 18             movzx eax, [ebp+018h]
%0000000000007f15 48                      dec eax
%0000000000007f16 01 45 f8                add dword [ebp-008h], eax
%0000000000007f19 48                      dec eax
%0000000000007f1a 8b 45 f8                mov eax, dword [ebp-008h]
%0000000000007f1d 0f b6 55 10             movzx edx, byte [ebp+010h]
%0000000000007f21 88 10                   mov byte [eax], dl
%0000000000007f23 eb 01                   jmp +001h (000007f26h)
%0000000000007f25 90                      nop
%0000000000007f26 48                      dec eax
%0000000000007f27 83 c4 10                add esp, byte 000000010h
%0000000000007f2a 5d                      pop ebp
%0000000000007f2b c3                      retn

When I discard the inc esp instruction at 0x7ef1,retn pops the return address correctly, but why is this instruction even there in the first place?

Try single-stepping the code from some earlier point. You are executing illegal instructions; the question is how you got to that address in the first place. There doesn't seem to be anything wrong with the C code, so the problem is most likely with the initialization that came before this, or perhaps something to do with your linker configuration. You'll need to show the code where the bug actually is - preferably a [mcve] from boot onward. — Nate Eldredge, Oct 30 '21 at 20:08
In particular, in your code that calls `KERNEL_MAIN()`, does it do something sensible after it returns? — Nate Eldredge, Oct 30 '21 at 20:10
Something tried to execute `f0 53` as an instruction. Based on the value of `EIP` (`00000083`) it looks like you've followed a bad pointer into the space where the real-mode IVT exists. Some virtual machine BIOSes use `f000:ff53` as the vector of unused interrupts, and it looks like you're trying to execute part of that byte sequence. Most likely you have some kind of stack corruption somewhere, but the given code isn't enough to figure out how. — sj95126, Oct 30 '21 at 20:12
Are you using a `stdcall` or other callee-pops calling convention with stack args? If so, caller and callee disagreeing on number of args will return with ESP pointing where the caller doesn't expect it. (That's why it's normal for toolchains that use such calling conventions to decorate function names with `foo@4` or whatever to indicate how many bytes it pops, to get link errors instead of runtime errors, so it's unlikely you have this problem.) — Peter Cordes, Oct 30 '21 at 20:37
From single-stepping the code I found that the parameters are not pushed onto the stack at all, but they get just moved into registers ecx, edx and eax. The Stack pointer was indeed messed up somehow, when the function was called, esp=0x2ffa7 and right before the return, esp=0x2ffa8. So one byte from the return address is somehow missing. Still trying to figure out why this happens though. — Mr.Gumba13, Oct 31 '21 at 08:00
That disassembly is 64-bit code. The single-byte `inc/dec` instructions were repurposed as the REX prefix in long mode. So you're trying to execute 64-bit code in 32-bit mode, and obviously it doesn't do the right thing. You need to fix your compiler options to build 32-bit code instead. (Or, if your kernel is supposed to be 64-bit, then switch into long mode before calling it.) — Nate Eldredge, Oct 31 '21 at 17:40

score 0 · Accepted Answer · answered Nov 04 '21 at 15:28

0

I found the solution when adding the compiler option:

gcc -mno-red-zone

It now works flawless, thanks!

answered Nov 04 '21 at 15:28

Mr.Gumba13

3
3

That can't have been the only change. Previously, you were in 32-bit mode, with the crash dump only showing 32-bit registers. But no standard 32-bit calling convention has a red-zone in the first place, only x86-64 System V does (out of all x86 32 and 64-bit ABIs). But yes, if you're building a 64-bit kernel, [you do need `-mno-red-zone`](https://stackoverflow.com/a/38043511/224132). (And probably `-mcmodel=kernel`, and also `-mno-sse` and other options to make the compiler limit itself to only touching integer regs.) – Peter Cordes Nov 05 '21 at 06:53

Why does my kernel crash when I add more arguments to a function?

1 Answers1