1

~ Identical question here, but sadly no useful answers.

I've used PlayStation Remote Play to stream live video from my PS5 to my computer. The stream is being shown in a window called 'PS Remote Play' on my computer. I would like to read frames from this window for eventual processing in opencv.

Using libraries mss and pygetwindow, I am able to capture this window correctly:

import cv2
import mss
import pygetwindow as gw
import numpy as np


def fetch_window_image(window_name):
    #
    window = gw.getWindowsWithTitle(window_name)[0]
    window_rect = {'left': window.left + 10, 'top': window.top + 45,
                   'width': window.width - 20, 'height': window.height - 60}
    with mss.mss() as sct:
        screenshot = sct.grab(window_rect)
        img = np.array(screenshot)
    img = cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)
    return img


def main():
    WINDOW_NAME = 'PS Remote Play'
    cv2.namedWindow('Computer Vision', cv2.WINDOW_NORMAL)
    while cv2.waitKey(1) != ord('q'):
        img = fetch_window_image(WINDOW_NAME)
        cv2.imshow('Computer Vision', img)


if __name__ == '__main__':
    main()

The problem is that this method is unable to capture it when the window is obscured by other windows (while remaining active). Being able to capture hidden windows is essential for my use case.

I came across the following code, which is able to capture hidden windows (source):

import win32gui
import win32ui
import win32con

w = 1920
h = 1080

hwnd = win32gui.FindWindow(None, 'PS Remote Play')
wDC = win32gui.GetWindowDC(hwnd)
dcObj = win32ui.CreateDCFromHandle(wDC)
cDC = dcObj.CreateCompatibleDC()
dataBitMap = win32ui.CreateBitmap()
dataBitMap.CreateCompatibleBitmap(dcObj, w, h)
cDC.SelectObject(dataBitMap)
cDC.BitBlt((0, 0), (w, h), dcObj, (0, 0), win32con.SRCCOPY)
dataBitMap.SaveBitmapFile(cDC, 'saved.bmp')

# Free Resources
dcObj.DeleteDC()
cDC.DeleteDC()
win32gui.ReleaseDC(hwnd, wDC)
win32gui.DeleteObject(dataBitMap.GetHandle())

Unfortunately, when used on the PS Remote Play window, it shows a black screen. I tried it on some other windows, and results were mixed. It was able to capture Spotify, Paint and the VSCode editor. It returned blank screens for PS Remote Play, Microsoft To Do, Whiteboard and Calendar. These behaviours remain the same whether visible or not.

Why is this happening, and how can we consistently capture hidden windows?

Nick_2440
  • 75
  • 1
  • 13
  • you're gonna have to keep that window open if you want your bot to play the game. – Christoph Rackwitz May 31 '23 at 13:13
  • It will be open. Just not visible because other windows will be covering it. And I don't plan on playing the game automatically, it's just to process image data from the screen. – Nick_2440 May 31 '23 at 13:31

2 Answers2

1

Quite a lot of games have multiple window modes, some of which blank the screen when not in focus to save on graphics a little. I am not sure if this setting exists in PS Remote Play, but try looking into the game settings and find an option that says "window mode" or "display mode" and select "windowed fullscreen" or "borderless" if you see it.

I encountered a similar issue quite a while ago with OBS not broadcasting a specific out-of-focus window correctly and this was how I solved it. Since your code works with other hidden windows, this is probably your issue too.

Squarish
  • 169
  • 6
  • A little testing found that I can see the PS Remote Play window when it is first loading up and connecting, but once the connection to my PS5 is made, then it becomes black. So the issue does in fact seem to be with the video feed being sent, not the app. I'm struggling to see any setting within the PS5 menus that are relevant to this though. – Nick_2440 Jun 03 '23 at 12:37
  • Another clue: when I use OBS to capture the window, using the capture method 'BitBlt' gives the black screen, while using 'Windows 10 (1903 and up)' shows it correctly. I have used a `BitBlt()` function in my code, so could this be the problem, and what should I replace it with? – Nick_2440 Jun 03 '23 at 12:56
  • I found this [old question](https://stackoverflow.com/questions/20601813/trouble-capturing-window-with-winapi-bitblt-for-some-applications-only) that seems to describe this issue. Problem is, pywin32 can only do BitBlt apparently. – Squarish Jun 03 '23 at 13:49
  • Putting my game in windowed fullscreen mode made it work with BitBlt so that would probably still be your easiest solution, but if you cannot, then you might have to use a different library that can handle OpenGL/DirectX or whatever accelerator your PS5 is using. – Squarish Jun 03 '23 at 13:51
  • I've found [this](https://github.com/BoboTiG/python-mss/issues/180) discussion about allowing the `mss` library (which works with hardware-accelerated graphics windows) to see obscured windows. They seem to be saying there is an easy fix, but it's not clear to me what to actually do - if you have time, would you be able to take a look? – Nick_2440 Jun 03 '23 at 16:18
1

Thanks to @wchill on Github for finding this solution (link)! Works on Windows, for windows which are both hardware accelerated and obscured by other windows.

import cv2
import numpy as np
from ctypes import windll
import win32gui
import win32ui


def capture_win_alt(window_name: str):
    # Adapted from https://stackoverflow.com/questions/19695214/screenshot-of-inactive-window-printwindow-win32gui

    windll.user32.SetProcessDPIAware()
    hwnd = win32gui.FindWindow(None, window_name)

    left, top, right, bottom = win32gui.GetClientRect(hwnd)
    w = right - left
    h = bottom - top

    hwnd_dc = win32gui.GetWindowDC(hwnd)
    mfc_dc = win32ui.CreateDCFromHandle(hwnd_dc)
    save_dc = mfc_dc.CreateCompatibleDC()
    bitmap = win32ui.CreateBitmap()
    bitmap.CreateCompatibleBitmap(mfc_dc, w, h)
    save_dc.SelectObject(bitmap)

    # If Special K is running, this number is 3. If not, 1
    result = windll.user32.PrintWindow(hwnd, save_dc.GetSafeHdc(), 3)

    bmpinfo = bitmap.GetInfo()
    bmpstr = bitmap.GetBitmapBits(True)

    img = np.frombuffer(bmpstr, dtype=np.uint8).reshape((bmpinfo["bmHeight"], bmpinfo["bmWidth"], 4))
    img = np.ascontiguousarray(img)[..., :-1]  # make image C_CONTIGUOUS and drop alpha channel

    if not result:  # result should be 1
        win32gui.DeleteObject(bitmap.GetHandle())
        save_dc.DeleteDC()
        mfc_dc.DeleteDC()
        win32gui.ReleaseDC(hwnd, hwnd_dc)
        raise RuntimeError(f"Unable to acquire screenshot! Result: {result}")

    return img


def main():

    WINDOW_NAME = "PS Remote Play"
    while cv2.waitKey(1) != ord('q'):
        screenshot = capture_win_alt(WINDOW_NAME)
        cv2.imshow('Computer Vision', screenshot)


if __name__ == '__main__':
    main()
Nick_2440
  • 75
  • 1
  • 13