Unity: screen space vs window space vs monitor space?

Question

I am trying to take a Ui object's screen space position and translate that to what I am calling 'monitor space'.

As far as I can tell, screen space, in Unity, is relative to the applications' window. That is, even if the app is not full screen, and moved around on your monitor, 0,0 will still be the lower left of the app window.

I need to translate one of those screen space values into the actual position within the user's monitor. This is especially important when considering that the user might have multiple monitors.

I am not finding anything to get this done, though.

I am hoping to find a platform agnostic solution, but if it must be Windows-only than I can make that work as well.

Any help on this would be greatly appreciated.

Thank you

Answer 1

Now after TEEBQNE's answer I also wanted to give it a shot using the native solution.

As mentioned this will be only for Windows PC Standalone and requires

• Unity's new Input System (see Quick Start)

• One of the solutions from Getting mouse position in c#

  For example if you want to use System.Windows.Forms then copy the according DLL from

   C:\Windows\Microsoft.NET\Framework64\v4.x.xx
  

  into your project under Assets/Plugins

  Then in code you can use

   using System.Windows.Forms;
  

If this is more efficient (or even works this way) I can't tell - only on the phone here - but I hope the idea gets clear ;)

So the idea is:

• store initial cursor position
• Set your cursor to certain positions of interest using WarpCursorPosition using Unity screen coordinates as input
• read out the resulting absolute monitor coordinates using the native stuff
• in the end reset the cursor to the original position

This might look somewhat like

using UnityEngine;
using UnityEngine.InputSystem;

public static class MonitorUtils
{
    // Store reference to main Camera (Camera.main is expensive)
    private static Camera _mainCamera;

    // persistent array to fetch rect corners
    // cheaper then everytime creating and throwing away a new array
    // especially when fetching them every frame
    private static readonly Vector3[] corners = new Vector3[4];

    // For getting the UI rect corners in Monitor pixel coordinates
    public static void GetMonitorRectCorners(this RectTransform rectTransform, Vector2Int[] output, bool isScreenSpaceCanvas = true, Camera camera = null)
    {
        // Lazy initialization of optional parameter
        if (!camera) camera = GetMainCamera();

        // Store initial mouse position
        var originalMousePosition = Mouse.current.position.ReadValue();

        // Get the four world space positions of your RectTtansform's corners
        // in the order bottom left, top left, top right, bottom right
        // See https://docs.unity3d.com/ScriptReference/RectTransform.GetWorldCorners.html
        rectTransform.GetWorldCorners(corners);

        // Iterate the four corners
        for (var i = 0; i < 4; i++)
        {
            if (!isScreenSpaceCanvas)
            {
                // Get the monitor position from the world position (see below)
                output[i] = WorldToMonitorPoint(corners[i], camera);
            }
            else
            {
                // Get the monitor position from the screen position (see below)
                output[i] = ScreenToMonitorPoint(corners[i], camera);
            }
        }

        // Restore mouse position
        Mouse.current.WarpCursorPosition(originalMousePosition);
    }
    
    // For getting a single Unity world space position in Monitor pixel coordinates
    public static Vector2Int WorldToMonitorPoint(Vector3 worldPoint, Camera camera = null)
    {
        // Lazy initialization of optional parameter
        if (!camera) camera = GetMainCamera();

        var screenPos = camera.WorldToScreenPoint(worldPoint);

        return ScreenToMonitorPoint(screenPos, camera);
    }

    // For getting a single Unity world space position in Monitor pixel coordinates
    public static Vector2Int ScreenToMonitorPoint(Vector3 screenPos, Camera camera = null)
    {
        // Lazy initialization of optional parameter
        if (!camera) camera = GetMainCamera();

        // Set the system cursor position there based on Unity screen space
        Mouse.current.WarpCursorPosition(screenPos);

        // Then get the actual system mouse position (see below)
        return GetSystemMousePosition();
    }

    // Get and store the main camera
    private static Camera GetMainCamera()
    {
        if (!_mainCamera) _mainCamera = Camera.main;

        return _mainCamera;
    }

    // Convert the system mouse position to Vector2Int for working 
    // with it in Unity
    private static Vector2Int GetSystemMousePosition()
    {
        var point = System.Windows.Forms.Cursor.Position;

        return new Vector2Int(point.X, point.Y);
    }
}

So you can either simply use

var monitorPosition = MonitorUtils.WorldToMonitorPoint(someUnityWorldPosition);
// or if you already have the `Camera` reference
//var monitorPosition = MonitorUtils.WorldToMonitorPoint(someUnityWorldPosition, someCamera);

or if you already have a screen space position like e.g. in a ScreenSpace Overlay canvas

var monitorPosition = MonitorUtils.ScreenToMonitorPoint(someUnityWorldPosition);
// or if you already have the `Camera` reference
//var monitorPosition = MonitorUtils.ScreenToMonitorPoint(someUnityWorldPosition, someCamera);

or you can get all four corners of a UI element at once using e.g.

 var monitorCorners = new Vector2Int [4];

 someRectTransform.GetMonitorRectCorners(monitorCorners, isScreenSpaceCanvas);
// or again if you already have a camera reference
//someRectTransform.GetMonitorRectCorners(monitorCorners, isScreenSpaceCanvas, someCamera);

---

Little example

public class Example : MonoBehaviour
{
    [Header("References")]
    [SerializeField] private Camera mainCamera;
    [SerializeField] private RectTransform _rectTransform;
    [SerializeField] private Canvas _canvas;

    [Header("Debugging")] 
    [SerializeField] private bool isScreenSpace;

    [Header("Output")] 
    [SerializeField] private Vector2Int bottomLeft;
    [SerializeField] private Vector2Int topLeft;
    [SerializeField] private Vector2Int topRight;
    [SerializeField] private Vector2Int bottomRight;

    private readonly Vector2Int[] _monitorPixelCornerCoordinates = new Vector2Int[4];

    private void Awake()
    {
        if (!mainCamera) mainCamera = Camera.main;
        if (!_canvas) _canvas = GetComponentInParent<Canvas>();
        isScreenSpace = _canvas.renderMode == RenderMode.ScreenSpaceOverlay;
    }

    private void Update()
    {
        if (Keyboard.current.spaceKey.isPressed)
        {
            _rectTransform.GetMonitorRectCorners(_monitorPixelCornerCoordinates, isScreenSpace);

            bottomLeft = _monitorPixelCornerCoordinates[0];
            topLeft = _monitorPixelCornerCoordinates[1];
            topRight = _monitorPixelCornerCoordinates[2];
            bottomRight = _monitorPixelCornerCoordinates[3];
        }
    }
}

You will see that moving your mouse each and every frame isn't a good idea though ^^

Now you can see the four corners being updated depending on the actual position on the screen.

Note: while Unity Screenspace is 0,0 at the bottom left in normal display pixels 0,0 is actually rather top-left. So you might need to invert these.

Answer 2

Alright first off - sorry for the late response just got back and was able to type up an answer.

From what I have found, this solution does not work in the editor and produces odd results on Mac with retina display. In the editor, the Screen and Display spaces appear to be exactly the same. There is probably a solution to fix this but I did not look into the specifics. As for Mac, for whatever reason, the internal resolution outputted is always half the actual resolution. I am not sure if this is just a retina display bug with Unity or a general Mac bug. I tested and ran this test script on both a Windows computer and Mac with a retina display. I have yet to test it on any mobile platform.

I do not know exactly what you would like to achieve with the values you wish to find, so I set up a demo scene displays the values instead of using them.

Here is the demo script:

using UnityEngine;
using System.Collections.Generic;
using UnityEngine.UI;

public class TestScript : MonoBehaviour
{
    [SerializeField] private RectTransform rect = null;
    [SerializeField] private List<Text> text = new List<Text>();
    [SerializeField] private Canvas parentCanvas = null;
    [SerializeField] private Camera mainCam = null;

    private void Start()
    {
        // determine the canvas mode of our UI object
        if (parentCanvas == null)
            parentCanvas = GetComponentInParent<Canvas>();

        // only need a camera in the case of camera space canvas
        if (parentCanvas.renderMode == RenderMode.ScreenSpaceCamera && mainCam == null)
            mainCam = Camera.main;

        // generate initial data points
        GenerateData();
    }

    /// <summary>
    /// Onclick of our button to test generating data when the object moves
    /// </summary>
    public void GenerateData()
    {
        // the anchored position is relative to screen space if the canvas is an overlay - if not, it will need to be converted to screen space based on our camera
        Vector3 screenPos = parentCanvas.renderMode == RenderMode.ScreenSpaceCamera ? mainCam.WorldToScreenPoint(transform.position) : rect.transform.position;

        // our object relative to screen position
        text[0].text = "Screen Pos: " + screenPos;

        // the dimensions of our screen (The current window that is rendering our game)
        text[1].text = "Screen dimensions: " + Screen.width + " " + Screen.height;

        // find our width / height normalized relative to the screen space dimensions
        float x = Mathf.Clamp01(screenPos.x / Screen.width);
        float y = Mathf.Clamp01(screenPos.y / Screen.height);

        // our normalized screen positions
        text[2].text = "Normalized Screen Pos: " + x + " " + y;

        // grab the dimensions of the main renderer - the current monitor our game is rendered on
#if UNITY_STANDALONE_OSX
        text[3].text = "Display dimensions: " + (Display.main.systemWidth * 2f) + " " + (Display.main.systemHeight * 2f);

        // now find the coordinates our the UI object transcribed from screen space normalized coordinates to our monitor / resolution coordinates
        text[4].text = "Display relative pos: " + (Display.main.systemWidth * x * 2f) + " " + (Display.main.systemHeight * y * 2f);
#else
        text[3].text = "Display dimensions: " + Display.main.systemWidth + " " + Display.main.systemHeight;

        // now find the coordinates our the UI object transcribed from screen space normalized coordinates to our monitor / resolution coordinates
        text[4].text = "Display relative pos: " + (Display.main.systemWidth * x) + " " + (Display.main.systemHeight * y);
#endif
    }

    /// <summary>
    /// Just for debugging - can be deleted
    /// </summary>
    private void Update()
    {
        if (Input.GetKey(KeyCode.A))
        {
            rect.anchoredPosition += new Vector2(-10f, 0f);
        }

        if (Input.GetKey(KeyCode.W))
        {
            rect.anchoredPosition += new Vector2(0f, 10f);
        }

        if (Input.GetKey(KeyCode.S))
        {
            rect.anchoredPosition += new Vector2(0f, -10f);
        }

        if (Input.GetKey(KeyCode.D))
        {
            rect.anchoredPosition += new Vector2(10f, 0f);
        }
    }
}

I accounted for the parent canvas being either Overlay or Camera mode and put in a check for an OSX build to adjust to the proper screen dimensions.

Here is a gif of the build on OSX. I set the window to be 1680x1050 and my computer's current resolution is 2880x1800. I had also test it on Windows but did not record it as the example looks nearly identical.

Let me know if you have more questions about the implementation or if there are issues with other platforms I did not test.

Edit: Just realized you want the screen space coordinate relative to the monitor space. I will correct the snippet in a little bit - in a meeting right now.

Edit2: After a bit more looking, it will not be easy to get the exact coordinates without the window being centered or getting the standalone window's position. I do not believe there is an easy way to get this information without a dll, so here is a implementation for mac and a solution for windows.

Currently, the solution I have will only get the screen position if the standalone player is windowed and centered on your screen. If the player is centered on the screen, I know that the center of my monitor is half the dimensions of its resolution, and know that the center point of my window matches up to this point. I can now get the bottom left corner of my window relative to my monitor and not a (0,0) coordinate. As the screen space has the bottom left corner at (0,0), you can now adjust the position to monitor space by adding the position of the newly calculated bottom left position.

Here is the new new GenerateData method:

/// <summary>
/// Onclick of our button to test generating data when the object moves
/// </summary>
public void GenerateData()
{
    // the anchored position is relative to screen space if the canvas is an overlay - if not, it will need to be converted to screen space based on our camera
    Vector3 screenPos = parentCanvas.renderMode == RenderMode.ScreenSpaceCamera ? mainCam.WorldToScreenPoint(transform.position) : rect.transform.position;

    // grab the display dimensions
    Vector2 displayDimensions;

    // bug or something with mac or retina display on mac where the main.system dimensions are half of what they actually are
#if UNITY_STANDALONE_OSX || UNITY_EDITOR_OSX
    displayDimensions = new Vector2(Display.main.systemWidth * 2f, Display.main.systemHeight * 2f);
#else
    displayDimensions = new Vector2(Display.main.systemWidth, Display.main.systemHeight);
#endif

    // the centerpoint of our display coordinates
    Vector2 displayCenter = new Vector2(displayDimensions.x / 2f, displayDimensions.y / 2f);

    // half our screen dimensions to find our screen space relative to monitor space
    Vector2 screenDimensionsHalf = new Vector2(Screen.width / 2f, Screen.height / 2f);

    // find the corners of our window relative to the monitor space
    Vector2[] displayCorners = new Vector2[] {
                                                    new Vector2(displayCenter.x - screenDimensionsHalf.x, displayCenter.y - screenDimensionsHalf.y),  // bottom left
                                                    new Vector2(displayCenter.x - screenDimensionsHalf.x, displayCenter.y + screenDimensionsHalf.y),  // top left
                                                    new Vector2(displayCenter.x + screenDimensionsHalf.x, displayCenter.y + screenDimensionsHalf.y),  // top right
                                                    new Vector2(displayCenter.x + screenDimensionsHalf.x, displayCenter.y - screenDimensionsHalf.y)   // bottom right
                                            };

    for (int z = 0; z < 4; ++z)
    {
        text[z].text = displayCorners[z].ToString();
    }

    // outputting our screen position relative to our monitor
    text[4].text = (new Vector2(screenPos.x, screenPos.y) + displayCorners[0]).ToString();
}

Once you are able to either get or set the windowed screen, you can properly re-orient the lower-left corner relative to the monitor dimensions or you can set the window back to the center point of your monitor. The above snippet would also work for a full-screen player. You would just need to determine how far off the aspect ratio of the player window is to your monitor, which allows you to find how large the black bars would be on the edges.

I assumed what you had wanted was straightforward but from what I can tell an OS-agnostic solution would be difficult. My above solution should work for any platform when the player is windowed if you can either get or set the standalone window position and for any platform that is full-screened with the theoretical approach I mentioned.

If you want more info on how to adjust the implementation for the full-screened window let me know.