-4

I want to be able to save a big file in seconds. Currently I can save 1GB in 4.8 sec (see next code):

import time
chunk="a"*1024*1024*1024; # 1GB 
t0=time.time();
open("test.txt", "w").write(chunk);
d=time.time()-t0;
print ("duration: %.2f s." % d);

duration: 4.80 s.

I am using a local disk Intel(R) Xeon(R) CPU E3-1505M v5, and RAM 64 GB. I get less than 209MB/sec of throughput.

How can I make 1GB in 1/sec or better!
Any suggestions are welcome. Thanks.

Universe196
  • 3
  • 1
  • 1
    I think that you should also take into account the time taken to create `chunk`. – quamrana Sep 08 '22 at 17:09
  • 3
    writing a text file in python is probably a very bad way of benchmarking your storage. What is the use case you actually care about? – Marcus Müller Sep 08 '22 at 17:09
  • Have you tried using a different language? Since python is kind of slow. This would be faster in C languages (C++, C, C#). Threading is not an option, as corrected to me by @MarcusMüller – NikkieDev Sep 08 '22 at 17:10
  • 1
    Also, you might need to calibrate your understanding of what an appropriate bandwidth is. 1GB/s of storage write rate is **hard** to achieve. And, your CPU and RAM don't even tell you anything about your storage medium, so it's quite clear you haven't tried to figure out your actual storage hardware's maximum write rates from datasheets so far! – Marcus Müller Sep 08 '22 at 17:10
  • 2
    @NikkieDev "threading" is not the answer to "writing my file isn't fast enough". Using more CPU cores, in the best case, has no overhead. It will not make IO any faster. You might do *something else* while waiting for IO to complete with multithreading, but nowadays, you'd implement that using asynchronous IO. – Marcus Müller Sep 08 '22 at 17:11
  • Its slightly faster in my machine using [Rust](https://rust-lang.org). In Python average 0.3s, in Rust .18s average - both assuming file doesn't already exist. – rv.kvetch Sep 08 '22 at 17:26

1 Answers1

-1

I would suggest to try using Rust.

My code:

use std::fs;
use std::time::Instant;

fn main() {
    let s = "a".repeat(1024 * 1024 * 1024);

    let start = Instant::now();
    fs::write("./test.txt", s).unwrap();
    println!("duration: {:.0?}", start.elapsed());
}

Don't forget to run with --release flag.

Example:

$ rm test.txt; ./target/release/main
duration: 179ms

Example ptyhon:

rm test.txt; python main.py
duration: 0.30 s.

Btw, I'm using an M1 Mac with a 32GB ram, in case anyone is curious :-)

rv.kvetch
  • 9,940
  • 3
  • 24
  • 53