read/write to a disk without a file system

Question

I would like to know if anybody has any experience writing data directly to disk without a file system - in a similar way that data would be written to a magnetic tape. In particular I would like to know if/how data is written in blocks, and whether a certain blocksize needs to be specified (like it does when writing to tape), and if there is a disk equivalent of a tape file mark, which separates the archives written to a tape.

We are creating a digital archive for over 1 PB of data, and we want redundancy built in to the system in as many levels as possible (by storing multiple copies using different storage media, and storage formats). Our current system works with tapes, and we have a mechanism for storing the block offset of each archive on each tape so we can restore it.

We'd like to extend the system to work with disk volumes without having to change much of the logic. Another advantage of not having a file system is that the solution would be portable across Operating Systems.

Note that the ability to browse the files on disk is not important in this application, since we are considering this for an archival copy of data which is not accessed independently. Also note that we would already have an index of the files stored in the application database, which we also write to the end of the tape/disk when it is almost full.

EDIT 27/05/2020: It seems that accessing the disk device as a raw/character device is what I'm looking for.

file access is usually more portable than block device drivers... but here's a starting point: https://stackoverflow.com/questions/3520459/direct-access-to-hard-disk-with-no-fs-from-c-program-on-linux — root, May 19 '20 at 02:53
@root I would like to use the character device drivers. Would this not be portable across OS's... and if not, why not? — swami, May 27 '20 at 08:07
On Windows it's something like `\\\\.\\PhysicalDrive3`. On Linux, it's `/dev/sda`. And if you're on EC2 then it's `/dev/xvda`` - so now you have to discover which is the device. In neither case these OSes expose these devices as character devices. You'll have OS-dependent limitations on the size of I/O you can do. Not to mention error handling. — root, May 27 '20 at 10:50
Also, why would you want a char device if your system has a mechanism to store block offsets etc.? — root, May 27 '20 at 10:53
If the only reason you want to avoid a FS is to be able to do block I/O then you can just do block I/O on files (`pread()`/`pwrite()`) or create a huge 1PB file and map it as a loop device. — root, May 27 '20 at 10:56

score 1 · Accepted Answer · answered Mar 07 '23 at 19:20

You can read/write to disk directly without going through the filesystem by opening the /dev/sda device directly, using lseek to seek to the block you want to read/write and calling read/write to do your read or write. Don't forget to close the device when you are done with it. In Linux everything is a file so the API you use is actually the same API you would use if you were writing to the file system. The only difference is that you open /dev/sda directly.

sample code:

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

#define BLOCK_SIZE 512

int main(int argc, char *argv[]) {
    int fd;
    off_t offset;
    ssize_t nwritten;

    // Open the device file for the disk you want to write to
    fd = open("/dev/sda", O_WRONLY);
    if (fd == -1) {
        perror("open");
        return 1;
    }

    // Seek to the appropriate block
    offset = BLOCK_SIZE * 100;
    if (lseek(fd, offset, SEEK_SET) == -1) {
        perror("lseek");
        return 1;
    }

    // Write the data to the block
    char data[BLOCK_SIZE] = "Hello, World!";
    nwritten = write(fd, data, BLOCK_SIZE);
    if (nwritten == -1) {
        perror("write");
        return 1;
    }

    printf("Wrote %zd bytes to block %ld\n", nwritten, offset/BLOCK_SIZE);

    // Close the device file
    if (close(fd) == -1) {
        perror("close");
        return 1;
    }

    return 0;
}

read/write to a disk without a file system

1 Answers1