0

This question may be a bit of a mouthful. I would like to access any windows application that a user launch that has some UI, and re-render those windows in a game engine (each window on its own texture). In essence, I am trying to cast every application running on my machine into a 3D world (onto their own Texture).

I am using Unity and found a workaround to obtain this behavior using WIN32 API, but it comes with major caveat:

  1. I use EnumWindows() to get the list of Windows - unfortunately lots of Windows without UI get iterated
  2. For each Window, a script containing the Window coordinate (using GetWindowRect()) is called
  3. Each script then calls GetForegroundWindow() to figure out if this Window is being manipulated with from the user. If so, it accesses the frame buffer by calling Graphics CopyFromScreen() and using the position returned from GetWindowRect()

Essentially, when the user interacts with a Window, I copy the part of the screen where the window is located, and when the user does not interact with an application Window, he essentially has a screenshot.

Issues are that:

  • EnumWindows() return windows that do not have UI (EDIT: solution here is pretty great and fixed this issue)
  • If a Window refreshes its content (like a video playing), I will not refresh in time
  • I have to call EnumWindows() over and over to figure out if a new Window has been created
  • It also breaks if part of the Window is located partly outside the screen
  • If two windows are on top of each other, it breaks

Is there a better way to access the rendering of every app on Windows, or is the partial screen capture the best solution for my case please?

Community
  • 1
  • 1
Jary316
  • 291
  • 1
  • 4
  • 15
  • For my first question (non UI windows returned), this solution was great: http://stackoverflow.com/questions/7277366/why-does-enumwindows-return-more-windows-than-i-expected – Jary316 Apr 30 '15 at 05:00
  • New windows: SetWindowsHookEx for a shell hook for HSHELL_WINDOWCREATED notifications or a CBT hook, also see http://stackoverflow.com/questions/5069104/fastest-method-of-screen-capturing – Alex K. Apr 30 '15 at 10:30
  • Forcing a window to render itself is not generally possible. It usually fails for windows, that are either obscured by other windows, or minimized. Even windows that are fully visible can simply call [SetWindowDisplayAffinity](https://msdn.microsoft.com/en-us/library/windows/desktop/dd375340.aspx), and prevent you from copying their content. – IInspectable Apr 30 '15 at 12:47
  • Thank you Alex, this is the big part I was missing! Thanks IInspectable. I assume this is the only way I can do it, knowing there are even more caveats. As a prototype, it seems to work decently well, so I might keep this implementation and just add the Windows Hook. Thanks! – Jary316 Apr 30 '15 at 17:44

0 Answers0