My apologies if this isn't the right exchange for this type of question. I was wondering if there are any resources on best practices and techniques for designing a custom full screen UI like that used on an ATM.
For example problems like supporting multiple resolutions, used fixed vs relative locations for text and buttons, etc.
I think that Qt Quick (aka QML) is the one that should suit your needs.
There are few good examples how QML works Here
QML is very effective and very easy to create. It can be used as standalone QML application, where business logic is done with javascript, or you can use it with C++ where you have very good control over it.
Also QML has very good support for different screen resolutions etc. because it is designed to be used in many different platforms.
I've just finished reading GUI Bloopers 2.0 by Jeff Johnson, any GUI designer for any application needs it.
