How to read a large file with function read() in C

Question

Aloha,

I'm new here, so please take it easy on me. I'm trying to read a file with function read() and then write() to a file or a file descriptor. My function successfully reads a file, but a problem occurs when I try to read a larger file(in my example size of 40,000 bytes).

I think that I must write a while loop, which will be reading until the end of a file, but I am stuck on the idea of how to..

(I open a file or file descriptor in main of the program)

My function( also convert binary input char data and writes to the ASCII) :

void function(int readFrom,int writeOn){
    char buffer[100];
    int x = read(readFrom, buffer, sizeof(buffer));
    int size= x/8;
    int i;
    for(i=0; i<size; i++){
        char temp[sizeof(int)-1];
        sprintf(temp,"%d",buffer[i];
        write(writeOn, temp, sizeof(temp));
    }
}

Possible duplicate of [Reading a large file using C (greater than 4GB) using read function, causing problems](http://stackoverflow.com/questions/11790822/reading-a-large-file-using-c-greater-than-4gb-using-read-function-causing-pro) — Hiren Jungi, Jan 25 '17 at 19:04
this statement: `int size= x/8;` uses integer math, so if the amount of bytes read is not a multiple of 8, some bytes will never be processed. Since 100 is not a multiple of 8, you can expect to miss outputting 4 bytes for every pass through the outer loop. Also, when the `read()` fails to input 100 bytes, the execution can be expected to lose upto 7 bytes for every pass through the outer loop. Much better to read 1024 bytes and use the returned value from `read()` as the count for the call to `write()` — user3629249, Jan 26 '17 at 13:51
@reny310: You can accept one of the answers by clicking on the grey checkmark below its score. — chqrlie, Feb 02 '17 at 07:14

Rafał Krypa · Accepted Answer · 2017-01-25T19:54:22.557

3

You need to check return value of functions read and write. They return the number of bytes read/written that may be less than the number that you passed as third argument. Both read and write must be done in a loop like:

int bytesRead = 0;

while (bytesRead < sizeof(buffer)) {
    int ret = read(readFrom, buffer + bytesRead, sizeof(buffer) - bytesRead);

    if (ret == 0)
        break; / * EOF */

    if (ret == -1) {
        /* Handle error */
    }

    bytesRead += ret;
}

edited Jan 25 '17 at 19:54

answered Jan 25 '17 at 19:08

Rafał Krypa

243
2
6

2

If you reach EOF before you fill the buffer, you're stuck in an infinite loop. – Petr Skocik Jan 25 '17 at 19:11

chqrlie · Answer 2 · 2017-01-26T23:37:58.933

You use sprintf() to convert characters from buffer into a very small buffer temp. On most current systems, int is 4 bytes, so your printf causes buffer overflows for char values greater than 99 (ASCII letter 'c'). Note that char can be signed by default, so negative values less than -99 will require 5 bytes for the string conversion: 3 digits, a minus sign and a null terminator.

You should make this buffer larger.

Furthermore, I don't understand why you only handle x/8 bytes from the buffer read by the read() function. The purpose of your function is obscure.

How to read a large file with function read() in C

2 Answers2