17

I want to write a simple TCP server using sockets in Python. The server should send the image to the connected client. The client should receive the image. But, for now on, the client receives only the portion of the image, and I can't even open it.

Server is multi-client using select, but its not the problem here. I think that the problem is with sending image.

I wanted "the protocol" to be very simple here.

SERVER                   CLIENT
               GET
       <----------------
              IMAGE
       ----------------->
      END OF COMMUNICATION

So the client can only send the "GET" message to server, and the server, after getting the "GET" string, should immediately send the whole image to client. That's it, communication is over.

server.py

#!/usr/bin/env python

import random
import socket, select
from time import gmtime, strftime

image = 'image.png'

HOST = '127.0.0.1'
PORT = 6666

connected_clients_sockets = []

server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen(10)

connected_clients_sockets.append(server_socket)

while True:

    read_sockets, write_sockets, error_sockets = select.select(connected_clients_sockets, [], [])

    for sock in read_sockets:

        if sock == server_socket:

            sockfd, client_address = server_socket.accept()
            connected_clients_sockets.append(sockfd)

        else:
            try:
                data = sock.recv(4096)
                bytes = open(image).read()

                if data:
                    sock.send(bytes)

            except:
                sock.close()
                connected_clients_sockets.remove(sock)
                continue

server_socket.close()

client.py

#!/usr/bin/env python

import socket
import sys

HOST = '127.0.0.1'
PORT = 6666

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = (HOST, PORT)
sock.connect(server_address)

try:

    sock.sendall("GET")

    while True:

        myfile = open('imagefromserv.png', 'w')

        while True:
            data = sock.recv(4096)
            if not data:
                break
            myfile.write(data)
        myfile.close()

finally:
    sock.close()

I'm using Python 2.7 on newest Ubuntu.

------------------------------------------------------------------------------------------------------------------------------------ EDIT ------------------------------------------------------------------------------------------------------------------------------------

Following advices given by one of the users in comments, I TRIED to implement a simple protocol:

CLIENT                                      SERVER
                      GET\r\n   
       ----------------------------------->
                      OK\r\n
       <----------------------------------- 
                   GET_SIZE\r\n
       ----------------------------------->
                    SIZE 1024\r\n
       <-----------------------------------
                   GET_IMG\r\n
       ----------------------------------->
                  IMG_DATA\r\r
       <-----------------------------------

Everything seems to work, but after the image transfer, my CPU is 100% busy, as top says. And ....

Server's output:

--GET--
--GET_SIZE--
--24518--
--GET_IMG--

Client's output:

--OK--
--SIZE 24518--
--24518--
4096
8192
12288
16384
20480
24523
Image received successfully

Indicates that the client received the image successfully. Is it ok now? I mean, I got the image from server, but I do not know, if I implemented the protocol correctly. Maybe something can be improved here?

client.py:

#!/usr/bin/env python

import socket
import sys

HOST = '127.0.0.1'
PORT = 6666

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = (HOST, PORT)
sock.connect(server_address)
fname = 'fromserver.png'

try:

    sock.sendall("GET\r\n")
    data = sock.recv(4096)

    if data:
        txt = data.strip()
        print '--%s--' % txt

        if txt == 'OK':

            sock.sendall("GET_SIZE\r\n")
            data = sock.recv(4096)

            if data:
                txt = data.strip()
                print '--%s--' % txt

                if txt.startswith('SIZE'):

                    tmp = txt.split()
                    size = int(tmp[1])

                    print '--%s--' % size

                    sock.sendall("GET_IMG\r\n")

                    myfile = open(fname, 'wb')

                    amount_received = 0
                    while amount_received < size:
                        data = sock.recv(4096)
                        if not data :
                            break
                        amount_received += len(data)
                        print amount_received

                        txt = data.strip('\r\n')

                        if 'EOF' in str(txt) :
                            print 'Image received successfully'
                            myfile.write(data)
                            myfile.close()
                        else :
                            myfile.write(data)
finally:
    sock.close()

server.py:

    #!/usr/bin/env python

import random
import socket, select
from time import gmtime, strftime

image = 'tux.png'

HOST = '127.0.0.1'
PORT = 6666

connected_clients_sockets = []

server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen(10)

connected_clients_sockets.append(server_socket)

while True:

    read_sockets, write_sockets, error_sockets = select.select(connected_clients_sockets, [], [])

    for sock in read_sockets:

        if sock == server_socket:

            sockfd, client_address = server_socket.accept()
            connected_clients_sockets.append(sockfd)

        else:
            try:
                data = sock.recv(4096)

                if data :

                    txt = data.strip()
                    print '--%s--'%txt

                    if txt == 'GET' :
                        sock.sendall('OK\r\n')

                    elif txt == 'GET_SIZE' :

                        with open ('tux.png','rb') as f1:
                            file_size = len(f1.read())
                            f1.seek(0)

                        print '--%s--'%file_size

                        file_size = '%s' % file_size
                        sock.sendall('SIZE %s\r\n' % file_size)

                    elif txt == 'GET_IMG' :
                        with open(image, 'rb') as fp:
                            image_data = fp.read()

                        msg = '%sEOF\r\r' % image_data
                        sock.sendall(msg)
                        print msg

            except:
                sock.close()
                connected_clients_sockets.remove(sock)
                continue

server_socket.close()

Or maybe I should rather do:

sock.sendall(image_data)
sock.sendall('EOF\r\n')

instead of:

msg = '%sEOF\r\n' % image_data
sock.sendall(msg)

in client?

yak
  • 3,770
  • 19
  • 60
  • 111
  • What is the protocol that you're using on top of TCP to send and receive files? If you haven't documented it, you need to -- otherwise, it's pretty much impossible to tell whether the client and server are doing the right thing. – David Schwartz Feb 21 '17 at 00:00
  • @DavidSchwartz: I only want to send and receive images using TCP sockets - that's all, without additional protocols. – yak Feb 21 '17 at 00:02
  • Then your client has a serious bug in that it sends the string "GET" which is not part of any image. (Honestly, I can tell you this from years of experience -- you really need to document your protocol precisely. How is the request delimited? How does the server mark the end of the data? How does the client identify it?) – David Schwartz Feb 21 '17 at 00:02
  • In server code, use `sock.sendall(bytes)` instead of `sock.send(bytes)` - that should recursively send whole image file – VenkatC Feb 21 '17 at 00:24
  • Most likely, it is the extra, outer `while True:`. The file is opened, the data is recieved, the file is closed, then it starts over again. The file is opened again and truncated, recv() is called but there is no more data, the file is closed. It should be an infinite loop, or just hang...that depends on if the server calls shutdown() to indicate no more data. – Mark Tolonen Feb 21 '17 at 02:50
  • @MarkTolonen: Changed the code as you suggested, but still cant open the image. – yak Feb 21 '17 at 12:05
  • How big is the image file? I tested your code and it works fine for me – VenkatC Feb 21 '17 at 14:49
  • @VenkatC: Here's the image I'm using: https://s7.postimg.org/u625v78xn/sandwich.png. And after sending, I can't open it, as I described. – yak Feb 21 '17 at 20:48
  • Works fine for me. You may want to verify file `sum` after receiving on client side – VenkatC Feb 23 '17 at 20:13
  • Latest code works fine for me too. Are you sure you can open `'./xkcd/sandwitch.png'` fine? – daphtdazz Feb 24 '17 at 15:16
  • @daphtdazz: Yes, I can open this file. Funny thing, however. The problem was with the image, I checked two other images and everything works now. Strange. Anyway, thank you. – yak Feb 24 '17 at 21:09
  • I had same problem a long time ago (was doing it in C) the problem was a CRC problem i suggest you try sending a text file first, then you can find out what is missing. BTW have you tried `myfile.flush()`, before `myfile.close()`, – Yonas Kassa Feb 27 '17 at 14:03

5 Answers5

12

You accidentally forgot to just use HTTP and Twisted.

Server:

from twisted.web.static import File
from twisted.web.resource import Resource
def resource():
    resource = Resource()
    resource.putChild(b"", File(u"xkcd/sandwich.png"))
    return resource

Client:

from filepath import FilePath
from twisted.internet.task import react
from treq import get, content

def main(reactor):
    d = get(b"http://localhost:8080/")
    d.addCallback(content)
    d.addCallback(FilePath(u"image.png").setContent)
    return d

react(main, [])

Server demo:

(everything) exarkun@baryon:/tmp/demo$ twist web --class server.resource
2017-02-23T21:32:14-0500 [-] Site starting on 8080
2017-02-23T21:32:14-0500 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site instance at 0x7fd1ef81a8c0>
2017-02-23T21:32:14-0500 [twisted.application.runner._runner.Runner#info] Starting reactor...
2017-02-23T21:33:01-0500 [twisted.python.log#info] "127.0.0.1" - - [24/Feb/2017:02:33:01 +0000] "GET / HTTP/1.1" 200 21489 "-" "-"
^C
2017-02-23T21:33:05-0500 [-] Received SIGINT, shutting down.
2017-02-23T21:33:05-0500 [-] (TCP Port 8080 Closed)
2017-02-23T21:33:05-0500 [twisted.web.server.Site#info] Stopping factory <twisted.web.server.Site instance at 0x7fd1ef81a8c0>
2017-02-23T21:33:05-0500 [-] Main loop terminated.
(everything) exarkun@baryon:/tmp/demo$ 

Client demo:

(everything) exarkun@baryon:/tmp/demo$ ls -l image.png
ls: cannot access 'image.png': No such file or directory
(everything) exarkun@baryon:/tmp/demo$ python client.py 
(everything) exarkun@baryon:/tmp/demo$ ls -l image.png 
-rwxr-xr-x 1 exarkun exarkun 21489 Feb 23 21:33 image.png
(everything) exarkun@baryon:/tmp/demo$

If you want to learn more about how select-loop driven networking is done, you can peruse the Twisted implementation.

Jean-Paul Calderone
  • 47,755
  • 6
  • 94
  • 122
  • I'm afraid that does not answer my question, since I want to use sockets only. – yak Feb 24 '17 at 16:57
  • Good news - HTTP is made out of sockets. – Jean-Paul Calderone Feb 24 '17 at 17:05
  • Yes, I know all this. I know how to use libraries designed for this task in Python. My question was about the sockets ONLY. – yak Feb 24 '17 at 20:20
  • 1
    There are so many problems with your `sockets`-module code that any answer short of "throw away all of this code and rewrite it after learning how to use sockets" would be misleading. I'm not trying to be mean. I'm trying to help. There's just too much wrong with the premise of your question to suggest a simple, incremental fix. – Jean-Paul Calderone Feb 24 '17 at 21:16
  • 1
    Sure, there are lots of ways to send data. Some of them are easier than others. Without a *good reason*, you probably shouldn't invent a brand new way to send files, though. Doing so isn't *quite* trivial and I don't see any indication in this question that a new way is needed (there's no discussion of how HTTP is unsuitable, for example). – Jean-Paul Calderone Mar 01 '17 at 13:50
5

Your client sends the string "GET". You only want to send and receive image data and "GET" is not image data.

You may have other bugs, it's hard to tell without understanding your protocol. For example, how does one side know what it has gotten all the image data?

David Schwartz
  • 179,497
  • 17
  • 214
  • 278
  • I simply thought that it will be enough for client to just send the GET, so when the server receives it, it will simply send the whole image. That's it, and I wanted to implement it that way. However, it seems, it does not work like that, since I'm unable to open the image. – yak Feb 21 '17 at 00:06
  • When the server receives "it"? So the only legal thing to send is precisely the string "GET"? And how is the end of the file marked? Please, *please* document your protocol. It is impossible to tell if the server is doing the right thing if you don't specify what the right thing actually is! – David Schwartz Feb 21 '17 at 00:07
3

You could read this: https://picamera.readthedocs.io/en/release-1.12/recipes1.html and learn a lot on how to send pictures overs sockets.'Basic Recipes' gives you everything you need.

user10547
  • 31
  • 2
3

The code below the EDIT section seems good to me. I completely agree with @David Schwartz. If you're looking to implement a protocol you have to take into consideration a whole lot of things regarding the protocol design. Ex:

  • if you send a command, how the server reacts
  • which commands are allowed
  • implement error codes for server responses, etc

For that you can read "TCP/IP and Linux Protocol Implementation" it is a great book for such matter.

In the other hand, you can stick with the basics and continue using TCP sockets,my humble suggestion is that you implement some mechanism for the client to know if it has the complete information, like:

  • send a hash of the data, check the hash in the client
  • cut the info in chunks and send the chunk quantity, then in the client check how many chunks were received, and re-request the image in case of failure
  • or all of the above
PROW
  • 182
  • 3
  • 10
-3

This is really strange. I tried two different images, and the code worked. The problem was with the image then.

yak
  • 3,770
  • 19
  • 60
  • 111
  • You totally missed the point! Your code can only work by luck, and sometimes it will ... until it doesn't. – David Schwartz Feb 27 '17 at 18:13
  • @DavidSchwartz: But why? I do not understand, please explain – yak Mar 01 '17 at 11:02
  • Read my comments on your question. You never designed a protocol so there's no way to know whether the protocol will work reliably. What would you review exactly? How can you tell if the code implements the protocol correctly if there is no protocol? – David Schwartz Mar 01 '17 at 13:55
  • @DavidSchwartz: I tried but still have problem with the image sending part (please see my edit). Can you help? – yak Mar 01 '17 at 16:00
  • @DavidSchwartz: And another question. If I use UDP instea of TCP, I will not need to design a protocol? Do I understand it right? – yak Mar 01 '17 at 17:15
  • **No.** How do you send a file using UDP? How does the receiver know when they have the whole file? How are dropped datagrams handled? UDP and TCP are network layer protocols, they need layers on top of them just like IP needs UDP and TCP (or something else) on top of it to be useful. Take a look at, for example, the protocol specification for [TFTP](https://tools.ietf.org/html/rfc1350), a file transfer protocol layered on top of UDP, or [FTP](https://www.ietf.org/rfc/rfc959.txt), a file transfer protocol layered on top of TCP. Neither TCP nor UDP are file transfer protocols. – David Schwartz Mar 01 '17 at 20:14
  • @Ok, thank you. I need to do some reading. But, is my last code good? Is it how I should implement this protocol? I mean it works good now, but I'm not sure if I understood you right. – yak Mar 02 '17 at 11:42
  • There's no way to know if your code is good or not because there's no specification for what it's supposed to do. How is the receiver supposed to know when it has the whole file? What is the sender supposed to send to request the file? How is the server supposed to know when it has the whole request? And so on. If you don't know what it's supposed to be doing, you can't say if it's doing the right thing or not. – David Schwartz Mar 02 '17 at 16:30
  • @DavidSchwartz: But I posted the protocol I want to implement ... so I think I know what I would like to do. – yak Mar 02 '17 at 18:03
  • You did not. If you did, then explain how the server knows when it has received the entire command and should proceed to send the file. Where's the part that documents what is a legal command and what isn't? – David Schwartz Mar 02 '17 at 18:14
  • @DavidSchwartz: Thank you for your patience. I really want to understand this. Ok, I updated the code again: https://gist.github.com/anonymous/dab25381d5ed50d6974bf2b022c2260d, and https://gist.github.com/anonymous/20f1a061e95435229e99d883ff16787e – yak Mar 02 '17 at 18:24