Copying a huge file using read(), write() and open() API's in linux

Question

I have been learning system programming in linux and I'm trying to copy a video using read() and write (). The problem I'm facing is that I cannot save the entire file into a buffer since its a huge file.

I thought I could loop it in since I was using write with append flag but then how would I use it with read?

This is my messed up code. I would appreciate any help:

int main() {

    int movie_rdfd = open("Suits.mp4", O_RDONLY); //fd for read
    off_t file_length = (int)(fseek(movie_rdfd, 0, SEEK_END));

    printf("This is fd for open: %d\n", movie_rdfd); //line to be deleted
    char* Save[fseek(movie_rdfd, 0, SEEK_END)];

    int C1 = read(movie_rdfd, Save, );

    printf("Result of Read (C1): %d\n", C1); //line to be deleted

    int movie_wrfd = open("Suits_Copy.mp4", O_WRONLY|O_CREAT, 0644); //fd for write

    printf("This is result of open: %d\n", movie_wrfd); //line to be deleted

    int C2 = write(movie_wrfd, Save, fseek(movie_rdfd, 0, SEEK_END));

    printf("Result of Read (C2): %d\n", C2); //line to be deleted

    close(movie_rdfd);
    close(movie_wrfd);

    return 0;
}

Also its showing segmentation fault when I try to find file size

Answer 1

The proper logic for copying a file in POSIX.1 systems, including Linux, is roughly

Open source file
Open target file
Repeat:
    Read a chunk of data from source
    Write that chunk to target
Until no more data to read
Close source file
Close target file

Proper error handling adds a significant amount of code, but I consider it a necessity, not an optional thing to be added afterwards if one has the time to do so.

(I am so strict in this, that I'd fail anyone who omits error checking, even if their program otherwise functioned properly. The reason is basic sanity: A tool that may blow up in your hands is not a tool, it is a bomb. There are enough bombs in the software world already, and we don't need more "programmers" who create those. What we need are reliable tools.)

Here is an example implementation with proper error checking:

#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>

#define  DEFAULT_CHUNK  262144  /* 256k */

int copy_file(const char *target, const char *source, const size_t chunk)
{
    const size_t size = (chunk > 0) ? chunk : DEFAULT_CHUNK;
    char        *data, *ptr, *end;
    ssize_t      bytes;
    int          ifd, ofd, err;

    /* NULL and empty file names are invalid. */
    if (!target || !*target || !source || !*source)
        return EINVAL;

    ifd = open(source, O_RDONLY);
    if (ifd == -1)
        return errno;

    /* Create output file; fail if it exists (O_EXCL): */
    ofd = open(target, O_WRONLY | O_CREAT | O_EXCL, 0666);
    if (ofd == -1) {
        err = errno;
        close(ifd);
        return err;
    }

    /* Allocate temporary data buffer. */
    data = malloc(size);
    if (!data) {
        close(ifd);
        close(ofd);
        /* Remove output file. */
        unlink(target);
        return ENOMEM;
    }

    /* Copy loop. */
    while (1) {

        /* Read a new chunk. */
        bytes = read(ifd, data, size);
        if (bytes < 0) {
            if (bytes == -1)
                err = errno;
            else
                err = EIO;
            free(data);
            close(ifd);
            close(ofd);
            unlink(target);
            return err;
        } else
        if (bytes == 0)
            break;

        /* Write that same chunk. */
        ptr = data;
        end = data + bytes;
        while (ptr < end) {

            bytes = write(ofd, ptr, (size_t)(end - ptr));
            if (bytes <= 0) {
                if (bytes == -1)
                    err = errno;
                else
                    err = EIO;
                free(data);
                close(ifd);
                close(ofd);
                unlink(target);
                return err;
            } else
                ptr += bytes;
        }
    }

    free(data);

    err = 0;
    if (close(ifd))
        err = EIO;
    if (close(ofd))
        err = EIO;
    if (err) {
        unlink(target);
        return err;
    }

    return 0;
}

The function takes the target file name (to be created), source file name (to be read from), and optionally the preferred chunk size. If you supply 0, the default chunk size is used. On current Linux hardware, 256k chunk size should reach maximum throughput; smaller chunk size may lead to slower copy operation on some (big and fast) systems.

The chunk size should be a power of two, or a small multiple of a large power of two. Because the chunk size is chosen by the caller, it is dynamically allocated using malloc()/free(). Note that it is explicitly freed in error cases.

Because the target file is always created -- the function will fail, returning EEXIST if the target file already exists --, it is removed ("unlinked") if an error occurs, so that no partial file is left over in error cases. (It is a common bug to forget to free dynamically allocated data in the error path; this is often called "leaking memory".)

The exact usage for open(), read(), write(), close(), and unlink() can be found at the Linux man pages.

write() returns the number of bytes written, or -1 if an error occurs. (Note that I explicitly treat 0 and all negative values smaller than -1 as I/O errors, because they should not normally occur.)

read() returns the number of bytes read, -1 if an error occurs, or 0 if there is no more data.

Both read() and write() may return a short count; i.e., less than was requested. (In Linux, this does not happen for normal files on most local filesystems, but only an idiot relies on the above function to be used on such files. Handling short counts isn't that complex, as you can see from the above code.)

If you wanted to add a progress meter, for example using a callback function, for example

void progress(const char *target, const char *source,
              const off_t completed, const off_t total);

then it would make sense to add an fstat(ifd, &info) call before the loop (with struct stat info; and off_t copied;, the latter counting the number of bytes copied). That call too may fail or report info.st_size == 0, if the source is e.g. a named pipe instead of a normal file. This means that the total parameter might be zero, in which case the progress meter would display only the progress in bytes (completed), with the remaining amount unknown.

Answer 2

Here are some critiques, and then how I'd do it:

This is good:

int movie_rdfd = open("Suits.mp4", O_RDONLY); //fd for read

This is, ummm, not so good:

off_t file_length = (int)(fseek(movie_rdfd, 0, SEEK_END));

fseek() is for stdio-based FILE * pointers opened with fopen(), not int file descriptors from open. To get the size of a file opened with open(), use fstat():

struct stat sb;
int rc = fstat( movie_rdfd, &sb );

Now you know how big the file is. But if it's a really big file, it's not going to fit into memory, so this is bad:

char* Save[fseek(movie_rdfd, 0, SEEK_END)];

That's bad in multiple ways, too - it should be char Save[], not char *. But either way, for a really large file, it's not going to work - it's too big to put on the stack as a local variable.

And you don't want to read the whole thing at once, anyway - it likely won't work as you'll likely get a partial read. Per the read standard:

The read() function shall attempt to read nbyte bytes from the file associated with the open file descriptor ...

RETURN VALUE

Upon successful completion, these functions shall return a non-negative integer indicating the number of bytes actually read. ...

Note that it says "shall attempt to read" and it returns "the number of bytes actually read". So you have to handle partial reads with a loop anyway.

Here's one really simple way to copy a file using open(), read(), and write() (note that it really should have some more error checking - for example, the write() results should be checked to be sure they match the number of bytes read):

#define BUFSIZE ( 32UL * 1024UL )

char buffer[ BUFSIZE ];
int in = open( nameOfInputFile, O_RDONLY );
int out = open( nameOfOutputFile, O_WRONLY | O_CREAT | O_TRUNC, 0644 );

// break loop explicitly when read fails or hits EOF
for ( ;; )
{
    ssize_t bytesRead = read( in, buffer, sizeof( buffer ) );
    if ( bytesRead <= 0 )
    {
        break;
    }

    write( out, buffer, bytesRead );
}

Note that you don't even need to know how big the file is.

There are a lot of things you can do to make this a bit faster - they're usually not worth it as the above code will likely run at about 90% of maximum IO rate on most systems.