AnimateWindow won't fit the bill.
In my opinion these programs work by
- Creating a transparent window when needed, while computing icons to display
- Creating the animation and displaying it on the transparent window device context
- Handling everything on the window via custom events.
What you need is
- a comprehensive knowledge of GDI32
- an equally comprehensive knowledge of those dirty tricks used to create animations programmatically.
I'm sorry, but I think that this is not the kind of topic that can briefly explained on a single answer :-(
Eventually, you need something that augments and extends Windows's native graphic capabilities. There are companies that did this before: IIRC Serif.com did write their own GDI replacement for their DTP programs, and I think that all of these companies that make desktop enhancements did the same (or at least, they know how to squeeze GDI32 capabilities)