Rust File Size is double compared with the real size. How to deal with it?

Question

I need to get large data form rust part. So, i need write file to save them . The size of data is 2^27, and every element is 256 bit. When i write them to the ssd, check the size of file, the size is double of value which i calcualte.

For the issue, i write "test" to a file 10 times with the 256 bit. According to the calculate of the size is 320 B, but the file size is 640 B.

I do not know, the other 320 B is what? The lines and data both right, how do i fix the issue in rust?

code as below:

use std::io::prelude::*;
use std::fs::File;

fn main() {
    let mut file = File::create("/home/test/1.txt").unwrap();
    let  s = b"de0b143e8f548ff5ec6d9e6e6e2ad07362d7cfbaac778561ba7da98dbb6eb4b";
    
    let mut  i = 0;
    while i < 10{
    file.write(s).unwrap();
    file.write(&"\n".as_bytes()).unwrap();
    i += 1;}
}

`s` is 63 characters long, plus the `\n` makes 64, times 10 is 640 bytes, sounds right to me. Are you sure you don't want to write binary data instead of hexadecimal strings? — Jmb, Mar 22 '22 at 08:57
Relevant: https://stackoverflow.com/questions/52987181/how-can-i-convert-a-hex-string-to-a-u8-slice — Jmb, Mar 22 '22 at 09:00
Byte strings don't do what you probably think they do. For example, `b"de"` is not a one-byte sequence containing 0xde, it's a two-byte sequence containing 0x64 (ASCII code for `d`) and 0x65 (ASCII code for `e`). To get the single-byte representation, you can use `\x`, i.e. `b"\xde"`. That would make your whole string `b"\xde\x0b\x14\x3e\x8f\x54\x8f\xf5\xec\x6d\x9e\x6e\x6e\x2a\xd0\x73\x62\xd7\xcf\xba\xac\x77\x85\x61\xba\x7d\xa9\x8d\xbb\x6e\xb4b"`. — user4815162342, Mar 22 '22 at 09:48

score 0 · Answer 1 · answered Mar 28 '22 at 12:26

As others in the comments already alluded to, it looks like your issue is related to bytestrings vs byte arrays.

Here's your example, modified to write the byte array:

use std::fs::File;
use std::io::prelude::*;

fn main() {
    let mut file = File::create("test.txt").unwrap();
    //let s = b"de0b143e8f548ff5ec6d9e6e6e2ad07362d7cfbaac778561ba7da98dbb6eb4b";
    let byte_str = vec![
        0xde, 0x0b, 0x14, 0x3e, 0x8f, 0x54, 0x8f, 0xf5, 0xec, 0x6d, 0x9e, 0x6e, 0x6e, 0x2a, 0xd0,
        0x73, 0x62, 0xd7, 0xcf, 0xba, 0xac, 0x77, 0x85, 0x61, 0xba, 0x7d, 0xa9, 0x8d, 0xbb, 0x6e,
        0xb4, 0xb0
    ];

    let mut i = 0;
    while i < 10 {
        file.write(&byte_str).unwrap();
        file.write(&"\n".as_bytes()).unwrap();
        i += 1;
    }
}

This produces a file that looks like this:

~> hexdump -C test.txt
00000000  de 0b 14 3e 8f 54 8f f5  ec 6d 9e 6e 6e 2a d0 73  |...>.T...m.nn*.s|
00000010  62 d7 cf ba ac 77 85 61  ba 7d a9 8d bb 6e b4 b0  |b....w.a.}...n..|
00000020  0a de 0b 14 3e 8f 54 8f  f5 ec 6d 9e 6e 6e 2a d0  |....>.T...m.nn*.|
00000030  73 62 d7 cf ba ac 77 85  61 ba 7d a9 8d bb 6e b4  |sb....w.a.}...n.|
00000040  b0 0a de 0b 14 3e 8f 54  8f f5 ec 6d 9e 6e 6e 2a  |.....>.T...m.nn*|
00000050  d0 73 62 d7 cf ba ac 77  85 61 ba 7d a9 8d bb 6e  |.sb....w.a.}...n|
00000060  b4 b0 0a de 0b 14 3e 8f  54 8f f5 ec 6d 9e 6e 6e  |......>.T...m.nn|
00000070  2a d0 73 62 d7 cf ba ac  77 85 61 ba 7d a9 8d bb  |*.sb....w.a.}...|
00000080  6e b4 b0 0a de 0b 14 3e  8f 54 8f f5 ec 6d 9e 6e  |n......>.T...m.n|
00000090  6e 2a d0 73 62 d7 cf ba  ac 77 85 61 ba 7d a9 8d  |n*.sb....w.a.}..|
000000a0  bb 6e b4 b0 0a de 0b 14  3e 8f 54 8f f5 ec 6d 9e  |.n......>.T...m.|
000000b0  6e 6e 2a d0 73 62 d7 cf  ba ac 77 85 61 ba 7d a9  |nn*.sb....w.a.}.|
000000c0  8d bb 6e b4 b0 0a de 0b  14 3e 8f 54 8f f5 ec 6d  |..n......>.T...m|
000000d0  9e 6e 6e 2a d0 73 62 d7  cf ba ac 77 85 61 ba 7d  |.nn*.sb....w.a.}|
000000e0  a9 8d bb 6e b4 b0 0a de  0b 14 3e 8f 54 8f f5 ec  |...n......>.T...|
000000f0  6d 9e 6e 6e 2a d0 73 62  d7 cf ba ac 77 85 61 ba  |m.nn*.sb....w.a.|
00000100  7d a9 8d bb 6e b4 b0 0a  de 0b 14 3e 8f 54 8f f5  |}...n......>.T..|
00000110  ec 6d 9e 6e 6e 2a d0 73  62 d7 cf ba ac 77 85 61  |.m.nn*.sb....w.a|
00000120  ba 7d a9 8d bb 6e b4 b0  0a de 0b 14 3e 8f 54 8f  |.}...n......>.T.|
00000130  f5 ec 6d 9e 6e 6e 2a d0  73 62 d7 cf ba ac 77 85  |..m.nn*.sb....w.|
00000140  61 ba 7d a9 8d bb 6e b4  b0 0a                    |a.}...n...|
0000014a

(Note: hexdump output shows a file's contents as raw binary and text. The first column is the "byte number, in hex", the next block is the raw binary of the file written in hex, and the section on the right is what that hex translates to as text)

Your original program produces a file that looks like this:

00000000  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000010  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
00000020  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
00000030  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000040  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000050  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
00000060  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
00000070  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000080  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000090  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
000000a0  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
000000b0  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
000000c0  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
000000d0  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
000000e0  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
000000f0  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000100  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000110  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
00000120  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
00000130  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000140  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000150  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
00000160  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
00000170  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000180  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000190  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
000001a0  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
000001b0  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
000001c0  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
000001d0  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
000001e0  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
000001f0  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000200  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000210  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
00000220  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
00000230  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000240  64 65 30 62 31 34 33 65  38 66 35 34 38 66 66 35  |de0b143e8f548ff5|
00000250  65 63 36 64 39 65 36 65  36 65 32 61 64 30 37 33  |ec6d9e6e6e2ad073|
00000260  36 32 64 37 63 66 62 61  61 63 37 37 38 35 36 31  |62d7cfbaac778561|
00000270  62 61 37 64 61 39 38 64  62 62 36 65 62 34 62 0a  |ba7da98dbb6eb4b.|
00000280

Since your data was given as a string, rust wrote text that looks like hex data, instead of, raw hex data.

Rust File Size is double compared with the real size. How to deal with it?

1 Answers1