python socket works fine when debugging line by line but not working on the full run

Question

I'm working on a project that involves transferring a file with a CRC checksum calculation for each chunk of a file for example here 40960 bytes. My issue is that everything works fine when I debug the code line by line, but when I run the code completely, I get different CRC checksums at the receiver side. I am guessing the issue is from the buffer.

The problem is from sender's side.

Any help would be much appreciated and please let me know how to do this task.

Thank you.

Sender Side:

import socket
import tqdm
import os
import crcmod

SEPARATOR = "<SEPARATOR>"
BUFFER_SIZE = 40960 # bytes

host = "10.218.105.192"
port = 5001

crc32 = crcmod.mkCrcFun(0x104c11db7, 0, False, 0xFFFFFFFF)

s = socket.socket()
print(f"[+] Connecting to {host}:{port}")
s.connect((host, port))
print("[+] Connected.")

#filename = input('Enter file name --> ')
filename="testfile"
if not os.path.exists(filename):
    print("file-doesn't-exist")
    s.close()
filesize = os.path.getsize(filename)
crc_of_file = open(filename, 'rb')
crc_d = crc_of_file.read()
crc = crc32(crc_d)
# send the filename and filesize
#s.send(f"{filename}{SEPARATOR}{filesize}{SEPARATOR}{crc}".encode())
# start sending the file
chunk_crc=[]
progress = tqdm.tqdm(range(filesize), f"Sending {filename}", unit="B", unit_scale=True, unit_divisor=1024)
with open(filename, "rb") as f:
    while True:
        # read the bytes from the file
        bytes_read = f.read(BUFFER_SIZE)
        if not bytes_read:
            # file transmitting is done
            break
        bytes_read += crc32(bytes_read).to_bytes(4, 'big')# we use sendall to assure transimission in
        # busy networks
        #s.send(f"{this_chunk_crc}\n".encode())
        s.sendall(bytes_read)
        # update the progress bar
        progress.update(len(bytes_read))
# close the socket
s.close()

Receiver Side:

import socket
import tqdm
import os
import crcmod
# device's IP address
SERVER_HOST = "0.0.0.0"
SERVER_PORT = 5001
# receive 40964 bytes each time which includes CRC checksum
BUFFER_SIZE = 40964


SEPARATOR = "<SEPARATOR>"
s = socket.socket()
s.bind((SERVER_HOST, SERVER_PORT))
s.listen(5)
print(f"[*] Listening as {SERVER_HOST}:{SERVER_PORT}")

client_socket, address = s.accept()
# if below code is executed, that means the sender is connected
print(f"[+] {address} is connected.")
# receive the file infos
# receive using client socket, not server socket
#received = client_socket.recv(BUFFER_SIZE).decode()
filename="testfile"
filesize=int("2367275")
#filename, filesize,crc = received.split(SEPARATOR)
# remove absolute path if there is
filename = os.path.basename(filename)
# convert to integer
filesize = int(filesize)
#print(f"[+] File CRC is {crc}")


crc32 = crcmod.mkCrcFun(0x104c11db7, 0, False, 0xFFFFFFFF)


# start receiving the file from the socket
# and writing to the file stream
progress = tqdm.tqdm(range(filesize), f"Receiving {filename}", unit="B", unit_scale=True, unit_divisor=1024)
#with open(filename, "wb") as f:
i=0
while True:
    # read bytes from the socket (receive)
    bytes_read = client_socket.recv(BUFFER_SIZE)
    if not bytes_read:
        # nothing is received
        # file transmitting is done
        break
    # write to the file the bytes we just received
    if (len(bytes_read) >= BUFFER_SIZE) :
        this_chunk_crc = crc32(bytes_read)
        print(this_chunk_crc)
        #partname=filename+"Part"+str(i)+" "+str(this_chunk_crc)
        with open(filename, "wb") as f:
            #bytes_read -= crc32(bytes_read).to_bytes(4, 'big')
            f.write(bytes_read)
    progress.update(len(bytes_read))


# close the client socket
client_socket.close()
# close the server socket
s.close()

the correct output should be something like (constant for that CRC definition should be 955982468 at every step as you can see and it is not related to the file)


Receiving testfile:   0%|          | 0.00/2.26M [00:00<?, ?B/s]955982468
Receiving testfile:   2%|▏         | 40.0k/2.26M [00:21<20:26, 1.90kB/s]955982468
Receiving testfile:   3%|▎         | 80.0k/2.26M [00:22<08:57, 4.26kB/s]955982468
Receiving testfile:   5%|▌         | 120k/2.26M [00:23<04:58, 7.52kB/s] 955982468
Receiving testfile:   7%|▋         | 160k/2.26M [00:23<03:08, 11.7kB/s]955982468
Receiving testfile:   9%|▊         | 200k/2.26M [00:24<02:08, 16.9kB/s]955982468
Receiving testfile:  10%|█         | 240k/2.26M [00:24<01:31, 23.2kB/s]955982468
Receiving testfile:  12%|█▏        | 280k/2.26M [00:25<01:08, 30.2kB/s]955982468
Receiving testfile:  14%|█▍        | 320k/2.26M [00:26<01:02, 32.9kB/s]955982468
Receiving testfile:  16%|█▌        | 360k/2.26M [00:26<00:49, 40.4kB/s]955982468
Receiving testfile:  17%|█▋        | 400k/2.26M [00:27<00:40, 47.9kB/s]955982468
Receiving testfile:  19%|█▉        | 440k/2.26M [00:27<00:35, 53.9kB/s]955982468
Receiving testfile:  21%|██        | 480k/2.26M [00:28<00:30, 60.5kB/s]955982468
Receiving testfile:  22%|██▏       | 520k/2.26M [00:28<00:27, 66.2kB/s]955982468
Receiving testfile:  24%|██▍       | 560k/2.26M [00:29<00:25, 70.3kB/s]955982468
Receiving testfile:  26%|██▌       | 600k/2.26M [00:30<00:26, 67.2kB/s]955982468
Receiving testfile:  28%|██▊       | 640k/2.26M [01:01<06:52, 4.15kB/s]955982468
Receiving testfile:  29%|██▉       | 680k/2.26M [01:01<04:45, 5.85kB/s]955982468
Receiving testfile:  31%|███       | 720k/2.26M [01:02<03:19, 8.17kB/s]955982468
Receiving testfile:  33%|███▎      | 760k/2.26M [01:02<02:23, 11.1kB/s]955982468
Receiving testfile:  35%|███▍      | 800k/2.26M [01:03<01:43, 15.0kB/s]955982468
Receiving testfile:  36%|███▋      | 840k/2.26M [01:03<01:15, 19.8kB/s]955982468
Receiving testfile:  38%|███▊      | 880k/2.26M [01:04<01:01, 23.7kB/s]955982468
Receiving testfile:  40%|███▉      | 920k/2.26M [01:05<00:47, 29.9kB/s]955982468
Receiving testfile:  42%|████▏     | 960k/2.26M [01:05<00:37, 37.2kB/s]955982468
Receiving testfile:  43%|████▎     | 0.98M/2.26M [01:06<00:30, 44.5kB/s]955982468
Receiving testfile:  45%|████▍     | 1.02M/2.26M [01:06<00:26, 48.7kB/s]955982468
Receiving testfile:  47%|████▋     | 1.05M/2.26M [01:07<00:22, 55.8kB/s]955982468
Receiving testfile:  48%|████▊     | 1.09M/2.26M [01:07<00:18, 65.4kB/s]955982468
Receiving testfile:  50%|█████     | 1.13M/2.26M [01:08<00:17, 68.5kB/s]955982468
Receiving testfile:  52%|█████▏    | 1.17M/2.26M [01:08<00:15, 71.5kB/s]955982468
Receiving testfile:  54%|█████▎    | 1.21M/2.26M [01:09<00:15, 72.5kB/s]955982468
Receiving testfile:  55%|█████▌    | 1.25M/2.26M [01:09<00:14, 73.8kB/s]955982468
Receiving testfile:  57%|█████▋    | 1.29M/2.26M [01:10<00:13, 76.1kB/s]955982468
Receiving testfile:  59%|█████▉    | 1.33M/2.26M [01:10<00:12, 76.4kB/s]955982468
Receiving testfile:  61%|██████    | 1.37M/2.26M [01:11<00:12, 76.6kB/s]955982468
Receiving testfile:  62%|██████▏   | 1.41M/2.26M [01:11<00:11, 77.4kB/s]955982468
Receiving testfile:  64%|██████▍   | 1.45M/2.26M [01:12<00:11, 76.6kB/s]955982468
Receiving testfile:  66%|██████▌   | 1.48M/2.26M [01:12<00:09, 84.2kB/s]955982468
Receiving testfile:  67%|██████▋   | 1.52M/2.26M [01:13<00:09, 81.2kB/s]955982468
Receiving testfile:  69%|██████▉   | 1.56M/2.26M [01:13<00:09, 78.5kB/s]955982468
Receiving testfile:  71%|███████   | 1.60M/2.26M [01:14<00:08, 77.3kB/s]955982468
Receiving testfile:  73%|███████▎  | 1.64M/2.26M [01:15<00:08, 76.2kB/s]955982468
Receiving testfile:  74%|███████▍  | 1.68M/2.26M [01:15<00:07, 85.2kB/s]955982468
Receiving testfile:  76%|███████▌  | 1.72M/2.26M [01:15<00:06, 81.0kB/s]955982468
Receiving testfile:  78%|███████▊  | 1.76M/2.26M [01:16<00:05, 87.8kB/s]955982468
Receiving testfile:  80%|███████▉  | 1.80M/2.26M [01:16<00:05, 84.3kB/s]955982468
Receiving testfile:  81%|████████▏ | 1.84M/2.26M [01:17<00:05, 80.5kB/s]955982468
Receiving testfile:  83%|████████▎ | 1.88M/2.26M [01:17<00:05, 78.7kB/s]955982468
Receiving testfile:  85%|████████▍ | 1.91M/2.26M [01:18<00:04, 79.6kB/s]955982468
Receiving testfile:  87%|████████▋ | 1.95M/2.26M [01:18<00:03, 84.7kB/s]955982468
Receiving testfile:  88%|████████▊ | 1.99M/2.26M [01:19<00:03, 79.6kB/s]955982468
Receiving testfile:  90%|████████▉ | 2.03M/2.26M [01:20<00:03, 76.8kB/s]955982468
Receiving testfile:  92%|█████████▏| 2.07M/2.26M [01:20<00:02, 81.9kB/s]955982468
Receiving testfile:  93%|█████████▎| 2.11M/2.26M [01:21<00:02, 62.6kB/s]955982468
Receiving testfile:  95%|█████████▌| 2.15M/2.26M [01:22<00:01, 63.4kB/s]955982468
Receiving testfile:  97%|█████████▋| 2.19M/2.26M [01:22<00:01, 70.8kB/s]955982468
Receiving testfile:  99%|█████████▊| 2.23M/2.26M [01:23<00:00, 69.7kB/s]955982468
Receiving testfile: 2.26MB [02:00, 19.7kB/s]

In short: you are assuming that `recv` will receive everything. This assumptions is wrong. With debugging it likely works for smaller files since the receiver is slower and thus all data are already accumulated in the receivers socket buffer and thus are available when doing the `recv`. Without debugging some data are still in flight or not transmitted at all and thus are not available for `recv`. — Steffen Ullrich, Apr 12 '22 at 05:24
@SteffenUllrich When we are defining a buffer size it will wait to receive the specified bytes. Am I right? (I am using the same file for both debugging and running and the file size is 2.26 Megabytes) — Arash, Apr 12 '22 at 06:22
@SteffenUllrich And could you please let me know how to resolve it? — Arash, Apr 12 '22 at 06:32
*"When we are defining a buffer size it will wait to receive the specified bytes. Am I right?"* - No. It will return once the first data are read, even if the specified size was larger. *" how to resolve it?"* - check the return code of `recv` and read again until you got all data you expected. — Steffen Ullrich, Apr 12 '22 at 07:27
@SteffenUllrich Thank you. My issue is that I need to read 40964 bytes (one chunk) at a time and calculate the CRC for that chunk. I do not want to perform this operation after receiving the entire file. What should I do? — Arash, Apr 12 '22 at 19:02
Call `recv` again and again until you get all the bytes you want. — Steffen Ullrich, Apr 12 '22 at 19:20
@SteffenUllrich Thank you and sorry to bother you. I used `if (len(bytes_read) >= BUFFER_SIZE) :` it is better now but still CRC numbers are not correct. I also updated the question — Arash, Apr 12 '22 at 19:39
Your initial read for name,size and crc assumes that it gets everything in a single `recv` - and not more. This assumption is false too since you send the file content immediately after the prefix, which means that you might get both within the `recv`. TCP is a byte stream, not a message protocol. — Steffen Ullrich, Apr 12 '22 at 20:25
@SteffenUllrich I removed the initial reading on the filename, file size, and CRC (just entered them manually at the receive side) but still readings are not correct at the `print(this_chunk_crc)`. I have no idea what else should I do. :( — Arash, Apr 12 '22 at 21:00
You are always doing a `bytes_read = client_socket.recv(BUFFER_SIZE)`, thus overwriting what you've read so far. You need instead append to what you've read so far. And you should also not read BUFFER_SIZE but only read as much data as you actually expect, which decreases if you get more and more data. — Steffen Ullrich, Apr 12 '22 at 21:55

python socket works fine when debugging line by line but not working on the full run

0 Answers0

Linked