I don't think it can be done that easily if the mouse is outside the main frame. That said, you can always do the following:
1) Start a timer in your main frame and poll it every 50 milliseconds (or whatever suits you);
2) Once you poll it in your OnTimer event handler, check the mouse position via wx.GetMousePosition() (this will be in screen coordinates);
3) In the same OnTimer method, get the screen position of your frame, via frame.GetScreenPosition();
4) Compare the mouse position with the frame position (maybe using an euclidean distance calculation, or whatever suits you). Then you set your frame transparency accordingly to this distance (remember to put it fully opaque if the mouse is inside the frame rectangle).
I did just it for the fun of it, it shouldn't take more than 5 minutes to implement.
Hope this helps.
Andrea.