Transfer contents of a folder over network by python

Question

I am facing a problem writing a program to send contents of a folder over the network by using Python. There are a lot of examples out there, all the examples I found are assuming the receiver side knew name of the file he want to receive. The program I am trying to do assuming that the receiver side agree to receive a files and there is no need to request a file by its name from the server. Once the connection established between the server and the client, the server start send all files inside particular folder to the client. Here is a image to show more explanation:example here

Here are some programs that do client server but they send one file and assume the receiver side knew files names, so the client should request a file by its name in order to receive it. Note: I apologies for English grammar mistakes.

https://www.youtube.com/watch?v=LJTaPaFGmM4

http://www.bogotobogo.com/python/python_network_programming_server_client_file_transfer.php

python socket file transfer

Here is best example I found:

Server side:

import sys

import socket

import os

workingdir = "/home/SomeFilesFolder"

host = ''
skServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skServer.bind((host, 1000))
skServer.listen(10)
print "Server Active"
bFileFound = 0

while True:
    Content, Address = skServer.accept()
    print Address
    sFileName = Content.recv(1024)
    for file in os.listdir(workingdir):
        if file == sFileName:
            bFileFound = 1
            break

    if bFileFound == 0:
        print sFileName + " Not Found On Server"

    else:
        print sFileName + " File Found"
        fUploadFile = open("files/" + sFileName, "rb")
        sRead = fUploadFile.read(1024)
        while sRead:
            Content.send(sRead)
            sRead = fUploadFile.read(1024)
        print "Sending Completed"
    break

Content.close()
skServer.close()

Client side:

import sys

import socket

skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("ip address", 1000))

sFileName = raw_input("Enter Filename to download from server : ")
sData = "Temp"

while True:
    skClient.send(sFileName)
    sData = skClient.recv(1024)
    fDownloadFile = open(sFileName, "wb")
    while sData:
        fDownloadFile.write(sData)
        sData = skClient.recv(1024)
    print "Download Completed"
    break

skClient.close()

if there is a way to eliminate this statement from the client side:

sFileName = raw_input("Enter Filename to download from server : ")

and make the server side send all files one by one without waiting for the client to pick a file.

Answer 1

Here's an example that recursively sends anything in the "server" subdirectory to a client. The client will save anything received in a "client" subdirectory. The server sends for each file:

1. The path and filename relative to the server subdirectory, UTF-8-encoded and terminated with a newline.
2. The file size in decimal as a UTF-8-encoded string terminated with a newline.
3. Exactly "file size" bytes of file data.

When all files are transmitted the server closes the connection.

server.py

from socket import *
import os

CHUNKSIZE = 1_000_000

sock = socket()
sock.bind(('',5000))
sock.listen(1)

while True:
    print('Waiting for a client...')
    client,address = sock.accept()
    print(f'Client joined from {address}')
    with client:
        for path,dirs,files in os.walk('server'):
            for file in files:
                filename = os.path.join(path,file)
                relpath = os.path.relpath(filename,'server')
                filesize = os.path.getsize(filename)

                print(f'Sending {relpath}')

                with open(filename,'rb') as f:
                    client.sendall(relpath.encode() + b'\n')
                    client.sendall(str(filesize).encode() + b'\n')

                    # Send the file in chunks so large files can be handled.
                    while True:
                        data = f.read(CHUNKSIZE)
                        if not data: break
                        client.sendall(data)
        print('Done.')

The client creates a "client" subdirectory and connects to the server. Until the server closes the connection, the client receives the path and filename, the file size, and the file contents and creates the file in the path under the "client" subdirectory.

client.py

from socket import *
import os

CHUNKSIZE = 1_000_000

# Make a directory for the received files.
os.makedirs('client',exist_ok=True)

sock = socket()
sock.connect(('localhost',5000))
with sock,sock.makefile('rb') as clientfile:
    while True:
        raw = clientfile.readline()
        if not raw: break # no more files, server closed connection.

        filename = raw.strip().decode()
        length = int(clientfile.readline())
        print(f'Downloading {filename}...\n  Expecting {length:,} bytes...',end='',flush=True)

        path = os.path.join('client',filename)
        os.makedirs(os.path.dirname(path),exist_ok=True)

        # Read the data in chunks so it can handle large files.
        with open(path,'wb') as f:
            while length:
                chunk = min(length,CHUNKSIZE)
                data = clientfile.read(chunk)
                if not data: break
                f.write(data)
                length -= len(data)
            else: # only runs if while doesn't break and length==0
                print('Complete')
                continue

        # socket was closed early.
        print('Incomplete')
        break 

Put any number of files and subdirectories under a "server" subdirectory in the same directory as server.py. Run the server, then in another terminal run client.py. A client subdirectory will be created and the files under "server" copied to it.

Answer 2

So... I've decided I've posted enough in comments and I might as well post a real answer. I see three ways to do this: push, pull, and indexing.

Push

Recall the HTTP protocol. The client asks for a file, the server locates it, and sends it. So get a list of all the files in a directory and send them all together. Better yet, tar them all together, zip them with some compression algorithm, and send that ONE file. This method is actually pretty much industry standard among Linux users.

Pull

I identifed this in the comments, but it works like this:

1. Client asks for directory
2. Server returns a text file containing the names of all the files.
3. Client asks for each file.

Index

This technique is the least mutable of the three. Keep an index of all the files in the directory, named INDEX.xml (funny enough, you could model the entire directory tree in xml.) your client will request the xml file, then walk the tree requesting other files.

Answer 3

you need to send os.listdir() by using json.dumps() and encode it as utf-8 at client side you need to decode and use json.loads() so that list will be transfer to client place sData = skClient.recv(1024) before sFileName = raw_input("Enter Filename to download from server : ") so that the server file list can be display you can find at here its a interesting tool https://github.com/manoharkakumani/mano