bsdtar file deliminator byte?

Question

I'm using bsdtar to pipe the contents of a .tar.gz file containing many (millions) xml files to stdout.

Currently the command I'm using is:

$ bsdtar -x -f <file.tar.gz> -O | ...

In the downstream program (the ellipsis in the above command) I need to chunk the output stream into each file. I have read the man page for bsdtar but couldn't see anyway to specify a file deliminator byte.

Currently I'm using this rust code. Which only works because each file has the same XML declaration line (which I discard).

struct FileIter<'a> {
    buf_reader: StdinLock<'a>
}

impl FileIter<'_> {
    fn next_file<'a>(&mut self, buf: &'a mut Vec<u8>) -> Option<&'a [u8]> {
        buf.clear();
        loop {
            match self.buf_reader.read_until(b'?', buf) {
                // This is the file deliminator
                // <?xml version="1.0" encoding="UTF-8"?>\n
                Ok(0) => { break None; }
                Ok(_) => {
                    let buf_len = buf.len();
                    if buf_len >= 37 {
                        if &buf[buf_len - 37..buf_len] ==
                            b"<?xml version=\"1.0\" encoding=\"UTF-8\"?" {
                            buf.truncate(buf_len - 37);
                            if buf.len() > 2 {
                                break Some(&buf[2..]);
                            }
                        }
                    }
                }
                Err(err) => {
                    println!("{:?}", err);
                    break None;
                }
            }
        }
    }
}

Is there an option in bsdtar to specify a file deliminator byte? Or is there a more generic way I could do this is in Rust that would work for all files?

Have you considered using a Rust native library instead? Like https://crates.io/crates/tar — Lambda Fairy, Dec 14 '20 at 04:46
I haven't actually! That's such an obvious answer, I just reached for what I knew. Thank you ! — James Schinner, Dec 14 '20 at 04:49

score 1 · Accepted Answer · answered Dec 14 '20 at 05:17

Using Lambda Fairy's comment, I was able to pretty much use the exact example from the tar crate documentation:

Cargo.toml

...

[dependencies]
tar = "0.4.30"
flate2 = "1.0.19"

main.rs

extern crate tar;

use std::fs::File;
use std::io::prelude::*;

use tar::Archive;
use flate2::read::GzDecoder;

fn main() {
    let path = "file.tar.gz";
    let tar_gz = File::open(path).unwrap();
    let tar = GzDecoder::new(tar_gz);
    let mut a = Archive::new(tar);

    for file in a.entries().unwrap() {
        // Make sure there wasn't an I/O error
        let mut file = file.unwrap();

        // Inspect metadata about the file
        println!("{:?}", file.header().path().unwrap());
        println!("{}", file.header().size().unwrap());

        // files implement the Read trait
        let mut s = String::new();
        file.read_to_string(&mut s).unwrap();
        println!("{}", s);
    }
}

bsdtar file deliminator byte?

1 Answers1

main.rs