That's right. You need something more global since a user will not have focus set onto a single application. Take a look at this CodeProject article/Codeplex project
Regardless of language and framework this is how applications such as instant messengers determine whether or not a user is AFK. It should be noted though that this can be used for nefarious purposes such as key loggers.
Abstract:
This class allows you to tap keyboard and mouse and/or to detect their
activity even when an application runs in the background or does not
have any user interface at all. This class raises common .NET events
with KeyEventArgs and MouseEventArgs, so you can easily retrieve any
information you need.