I am developing a document based application for Mac OS X. It's a kind of media player, but instead of playing audio or video files it is supposed to open text-files containing meta-data specifying OpenGL animations. I would like to mimic Apples QuickTime X window style. This means, i have to do all the window drawings myself, because Cocoa has no appropriate window style.
There is one thing which gives me headaches: The rounded corners usually to be found on Mac OS X windows. I tried using the borderless window mask and working some CGS magic - there are some private Apple headers which allow window shaping, but they are of course undocumented. I was able to cut rectangular holes in my windows edges, but i couldn't figure out how Apple achieves rounded corners.
Creating a transparent window and drawing the frame myself does not work, because an OpenGL viewport is always rectangular, and the only way to change it is to turn on NSOpenGLCPSurfaceOpacity for alpha transparency and using the stencil buffer or shaders to cut out the edges, which seems like a hell of a lot of overhead.
If i put an OpenGLView into a standard Cocoa window with titlebar, the bottom edges are rounded. It seems this is happening at the NSThemeFrame stage of the view hierarchy. Any ideas how this is done?