3

Is there a way to use std::io::Write to write a stream directly to memory, without either pre-allocating the final at the outset, or potentially re-allocating the entire array while writing?

While Vec implements Write, this requires contiguous memory and I assume may re-allocate as with regular vector resizing.

Does Rust's stdlib support something like this, or would it need to be implemented?

I want to write to memory which has the potential to go into multiple GiB. It's possible the memory is fragmented and it's possible that a contiguous region of the size I want isn't available. Instead, it's quite efficient to allocate many chunks (probably fixed power-of-2 size), and allocate more chunks as they're needed.

Once finished you could read the data back, compress the chunks in memory, cache some to disk, binary-diff with previous writes to de-duplicate chunks... etc.

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
ideasman42
  • 42,413
  • 44
  • 197
  • 320
  • What about `std::io::BufWriter`? – ljedrz Feb 10 '17 at 13:45
  • @ljedrz AFAICS this is just an intermediate buffer that wraps another writer (it's only storing many small writes in a buffer before flushing, but isn't intended to be the target you write to)... unless I misunderstood the docs. – ideasman42 Feb 10 '17 at 13:47
  • Do you mean you want to allocate a bunch of smaller buffers and then do one combine/copy into a big buffer at the end? – Chris Emerson Feb 10 '17 at 13:47
  • @Chris Emerson, yes on a bunch of smaller buffers, and *maybe* combine at the end. But not necessarily. Could do other operations on it too (in memory compression, keep as-is for re-use... whatever). – ideasman42 Feb 10 '17 at 13:49
  • 2
    The documentation doesn't list any obvious implementations of `std::io::Write` that do what you want, so I think you'd need to implement it. – Chris Emerson Feb 10 '17 at 13:56
  • Suspected this, no big-deal, just wondered if I was missing something obvious, or if it was implemented in some other part of stdlib. – ideasman42 Feb 10 '17 at 13:57
  • 1
    This seems like a rather specialized requirement (where details can matter, to boot). In addition to that, it is quite easy to implement the `Write` trait to do what you want, e.g. using a `Vec>`. – user4815162342 Feb 10 '17 at 13:59
  • note that system malloc or mmap (but not necessarily jemalloc) should be cheap. You can reserve a lot of address space in advance and it won't actually need the memory until you touch those pages. so either a crate providing `Write` over mmaped-memory or trying `Vec` with system malloc might work. Of course that requires 64bit systems and probably tempfile-backed memory to avoid overcommit problems. – the8472 Feb 10 '17 at 15:55
  • Try Cursor, like suggested in https://stackoverflow.com/a/41069910/895727 – clonejo Jul 02 '23 at 18:05

1 Answers1

0

There is no such support in the Rust 1.15.1 standard library.

Once you've implemented such a data structure that handles being created and avoids whatever memory fragmentation issues you have, implementing Write should be straightforward.

If you have memory fragmentation issues on a 64-bit machine (16 Exabytes) or the smaller 48-bit addressing (256 terabytes), I would recommend rechecking your solution for appropriateness to the problem domain.

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366