At the risk of gross oversimplification, the only ways to enter kernel mode are though an exception (trap or fault) or an interrupt.
For system call, application needs to trigger a trap. Invariably, the system call has is a wrapper function that possibly does some validation, loads the parameters into the appropriate register (maybe set up a stack from). Then it executes an instruction that causes a trap
TRAP #100 ; call the 100th exception handler
After that, the processor enters kernel mode. There is will validate the parameters then execute the kernel service. At completion it will return using an insertion along the lines of this:
REI ; return from exception or interrupt
The the processor goes back to user mode.
open () may or may not be an actual system service. That is, one of the wrappers I just described. That depends upon the system. It is possible that open () could be a library function that calls multiple system services, in which case the processor would go into kernel mode and back multiple times.
If open () is a system service, the you probably go into kernel mode and back just once.
The open function, regardless of whether it is a system service or not, will invariably return the value in a register. Storing the value is a simple move instruction.