Freeze on sending large file using QTcpSocket

Question

I'm trying to send and receive some large files (at least 16GB) from a PC to another. The application freeze when the client application received around 2GB and it consume almost all of my memory (also used up around 2GB of memory). I do not have this problems with the serverside.

Here is server code that send file

clock_t startTime = clock();
QTcpSocket *clientConnection = tcpServer->nextPendingConnection();
connect(clientConnection, SIGNAL(disconnected()),
            clientConnection, SLOT(deleteLater()));

QString sourceFileName("/path/to/source/sourcefile.big");
QByteArray baFName=sourceFileName.toLocal8Bit();
char* c_fileName=baFName.data();

char buf[BUFSIZ];
size_t size;
unsigned long long sendSize=0;
int source = open64(c_fileName, O_RDONLY, 0);
unsigned long long loopCount=0;
while ((size = read(source, buf, BUFSIZ)) > 0) {
    sendSize=clientConnection->write(buf, size);
    clientConnection->waitForBytesWritten();
    if(sendSize< size) {
       qWarning("transmit error!");
    }
    qDebug() << "Loop #" << ++loopCount << " send data: " << sendSize;

}

qDebug() << double( clock() - startTime ) / (double)CLOCKS_PER_SEC<< " seconds.";

clientConnection->disconnectFromHost();

The client side of the application already know how big the file it received and here are the code that received the file and write it to disk

clock_t startTime = clock();

QString sourceFileName("/path/to/target/targetfile.big");
unsigned long long targetSize=16447314864ULL;

unsigned long long loopCount=(targetSize / 8192ULL) + ( targetSize % 8192ULL > 0 ? 1 : 0);
QByteArray baFName=sourceFileName.toLocal8Bit();
char* c_fileName=baFName.data();

char buf[BUFSIZ];
size_t size;
unsigned long long sendSize=0;
int dest = open64(c_fileName, O_WRONLY | O_CREAT, 0644);

while (loopCount){
    if (tcpSocket->waitForReadyRead()){
        size=tcpSocket->read(buf, 8192);
        write(dest, buf, size);
        qDebug() << "Loop #" << loopCount << " receive data: " << size;
        loopCount--;
    }
}

qDebug() << double( clock() - startTime ) / (double)CLOCKS_PER_SEC<< " seconds.";

I'm using ubuntu 14.04 if it matter.

score 2 · Answer 1 · answered May 27 '14 at 01:03

You should start by getting rid of loopCount. Compare the bytes received with the number of bytes to receive.

You will also want to make use of tcpSocket->bytesAvailable()

You may want to try this code: it may give you a general idea.

unsigned long long totalBytesRead = 0;
while (totalBytesRead < targetSize) {
    if (tcpSocket->waitForReadyRead()) {
            unsigned long long bytesAvailable = tcpSocket->bytesAvailable();
            char buf[bytesAvailable];
            totalBytesRead += bytesAvailable;
            write(dest, buf, size);
            qDebug() << "Loop #" << loopCount << " receive data: " << size;
    }
}

Also, since you're writing QT code, you may want to use QFile. It will also save you the trouble of having this:

char* c_fileName=baFName.data();

As you can have this instead

QFile file("/path/to/target/targetfile.big");

score 2 · Answer 2 · answered May 27 '14 at 06:13

The best approach is to receive data in an asynchronous mode by connecting the readyRead() signal of the socket to a slot :

connect( tcpSocket, SIGNAL(readyRead()),
        this, SLOT(tcpReady()) );

From the Qt documentation :

This signal is emitted once every time new data is available for reading from the device. It will only be emitted again once new data is available, such as when a new payload of network data has arrived on your network socket, or when a new block of data has been appended to your device.

You can read data in the slot which is connected to readyRead signal :

void MyClass::tcpReady()
{
    unsigned long long bytesAvailable = tcpSocket->bytesAvailable();
    char buf[bytesAvailable];
    totalBytesRead += bytesAvailable;
    write(dest, buf, size);
    qDebug() << "Loop #" << loopCount << " receive data: " << size;

}

score 0 · Accepted Answer · answered May 27 '14 at 09:33

I've managed to resolved the freeze and memory hogging thing of the client side. It turns out that waitForReadyRead() is the cause of it.

I just remove it and check the output of tcpSocket->read. If it's zero, then i put a delay of several ms so that it is not becoming a processor hog this time.

I wonder if the serverside can be faster if i remove the waitForByteWritten(). I have tried to check the clientConnection->bytesToWrite() and limit it to 1GB, but it turns out the number isn't going down. Is there any other method that I could to use to replace waitForByteWritten() in synchronous way?

also, i already check the output of clientconnection->write but it always return the same as size, never -1. — Ste, May 27 '14 at 09:40

Freeze on sending large file using QTcpSocket

3 Answers3