Read at position relative to end of the file via IoUring

Question

How can I read at the position relative to end of the file via IoUring? I have a file that contains the metadata in the end of the file because of single pass writing. Now, I want to read the file via IoUring for efficiency, therefore I have to read the metadata in the first step.

Read it via blocking IO is simple, we can use lseek with SEEK_END to seek to the position(file.seek(SeekFrom::End(-meta_siza)) in rust) and then read from it. However, after some search, I found IoUring does not have an operation named seek. So maybe we can use the pos = file.read_at(file_size - meta_size) to achieve it. But how can we get the file size? Using stat syscall may cause blocking...

`io_uring::opcode::Statx` is equivalent to the `statx()` system-call. It fills a `statx` struct that has a `stx_size` member. (I have not tried myself) — prog-fh, Mar 08 '23 at 10:04

prog-fh · Accepted Answer · 2023-03-08T12:06:47.243

This example was for me the occasion to discover io-uring.

One solution is to call statx() with AT_FDCWD as file-descriptor in order to state that the path is relative to the current directory. The path provided to this call is supposed to be null-terminated, hence the conversion into CString.

Another solution is to call statx() with an already open file-descriptor, an empty null-terminated string as filename and the AT_EMPTY_PATH flag.

Note that this example is not representative of a normal usage because it does not take any advantage of the asynchronous nature of the API; we submit and wait for every single operation in a synchronous manner.

// io-uring = ">=0"
// libc = ">=0"

use std::os::unix::io::AsRawFd;

type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;

fn file_size_by_name(
    ring: &mut io_uring::IoUring,
    filename: &str,
) -> Result<usize> {
    let user_data = 0xdead;
    let filename = std::ffi::CString::new(filename)?;
    let mut statxbuf = std::mem::MaybeUninit::<libc::statx>::zeroed();
    let entry = io_uring::opcode::Statx::new(
        io_uring::types::Fd(libc::AT_FDCWD),
        filename.as_ptr(),
        statxbuf.as_mut_ptr() as *mut _,
    )
    .build()
    .user_data(user_data);
    unsafe {
        ring.submission().push(&entry)?;
    }
    ring.submit_and_wait(1)?;
    let cqe = ring.completion().next().ok_or("no statx entry")?;
    if cqe.user_data() != user_data {
        Err("invalid user_data")?
    } else if cqe.result() < 0 {
        Err(format!("statx error: {}", cqe.result()))?
    } else {
        Ok(unsafe { statxbuf.assume_init_ref() }.stx_size as usize)
    }
}

fn file_size(
    ring: &mut io_uring::IoUring,
    fd: &std::fs::File,
) -> Result<usize> {
    let user_data = 0xdead;
    let mut statxbuf = std::mem::MaybeUninit::<libc::statx>::zeroed();
    let entry = io_uring::opcode::Statx::new(
        io_uring::types::Fd(fd.as_raw_fd()),
        [0i8].as_ptr(),
        statxbuf.as_mut_ptr() as *mut _,
    )
    .flags(libc::AT_EMPTY_PATH)
    .build()
    .user_data(user_data);
    unsafe {
        ring.submission().push(&entry)?;
    }
    ring.submit_and_wait(1)?;
    let cqe = ring.completion().next().ok_or("no statx entry")?;
    if cqe.user_data() != user_data {
        Err("invalid user_data")?
    } else if cqe.result() < 0 {
        Err(format!("statx error: {}", cqe.result()))?
    } else {
        Ok(unsafe { statxbuf.assume_init_ref() }.stx_size as usize)
    }
}

fn read_bytes(
    ring: &mut io_uring::IoUring,
    fd: &std::fs::File,
    offset: usize,
    buffer: &mut [u8],
) -> Result<usize> {
    let user_data = 0xbeef;
    let entry = io_uring::opcode::Read::new(
        io_uring::types::Fd(fd.as_raw_fd()),
        buffer.as_mut_ptr(),
        buffer.len() as _,
    )
    .offset64(offset as _)
    .build()
    .user_data(user_data);
    unsafe {
        ring.submission().push(&entry)?;
    }
    ring.submit_and_wait(1)?;
    let cqe = ring.completion().next().ok_or("no read entry")?;
    if cqe.user_data() != user_data {
        Err("invalid user_data")?
    } else if cqe.result() < 0 {
        Err(format!("read error: {}", cqe.result()))?
    } else {
        Ok(cqe.result() as usize)
    }
}

fn main() -> Result<()> {
    let mut ring = io_uring::IoUring::new(8)?;

    let filename = "Cargo.lock";
    let fd = std::fs::File::open(filename)?;

    let size = file_size(&mut ring, &fd)?;
    println!("{:?} contains {} bytes", filename, size);
    println!(
        "by name: {:?} contains {} bytes",
        filename,
        file_size_by_name(&mut ring, filename)?
    );

    let mut meta = [0; 10];
    let amount = read_bytes(&mut ring, &fd, size - meta.len(), &mut meta)?;
    println!("last {} bytes: {:?}", amount, &meta[..amount]);

    Ok(())
}
/*
"Cargo.lock" contains 811 bytes
by name: "Cargo.lock" contains 811 bytes
last 10 bytes: [34, 108, 105, 98, 99, 34, 44, 10, 93, 10]
*/

Read at position relative to end of the file via IoUring

1 Answers1