Can `dat` protocol efficiently support live streaming of video?

Question

I would like to be able to live stream video (or any other file that is large and continuously modified/appended) via dat.

Here it says,

The dat:// protocol doesn't support partial updates at the file-level, which means that with multiple records in a single file, every time a user adds a record, anyone who follows that user must sync and re-download the entire file. As the file continues to grow, performance will degrade. Putting each record in an individual file is much more efficient: when a record is created, peers in the network will only download the newly-created file.

However, it also says here that dat uses Rabin fingerprinting to create deterministic chunks of files, so presumably a dat client would be able to easily identify the chunks that it has already downloaded by their hash, and should therefore be able to only download the latest final chunk of the file, if that is the only part that has changed.

And also here in the faq, it says:

The type of Merkle tree used by Dat lets peers compare which pieces of a specific version of a dataset they each have and efficiently exchange the deltas to complete a full sync.

There is hypervision, but from my rudimentary understanding of how it works, it looks like it saves it's own "bundle.js" file for the video data, I'm not sure how it achieves streaming, but this is not exactly the same as what I'm trying to achieve, which is being able to efficiently stream an arbitrary large and expanding file, for example a .ts or .mkv video stream.

So, my question is - is efficient live-streaming of video (ie without redownloading already-downloaded chunks) something that is simply currently not supported and could be added in future, or is that something that is inherently unachievable using the dat protocol?

Answer 1

In short, the low-level hypercore protocol that Dat is built on top of should work well for video and other "soft realtime" streaming uses. However, the hyperdrive file/directory abstraction that Dat (the application) builds upon does not currently work well for these use cases. There is nothing blocking hyperdrive from working well with a single "arbitrary large and expanding file", but it has not been optimized for that specific use case.

---

As far as I know, all the current video streaming prototypes work by encoding video content directly into hypercore, not in a hyperdrive "files and directories" abstraction. This is sort of like the difference between writing raw bytes to a hard disk instead of using a file system. P2P video and audio streaming were explicit design goals for hypercore. Note that there may or may not be direct mappings to existing file formats or streaming protocols; the hypercore abstraction is presented as a stream of byte chunks, each capped at about a megabyte.

As a small detail, the dat/hypercore protocol and on-disk formats do not specify any particular "chunking" mechanism. Rabin-chunking has been experimented, but by default almost all clients use fixed-size chunking instead for simplicity and speed (which isn't to imply that it isn't possible to implement performant locality-sensitive chunking in the future). In theory the clients will be able to detect duplicate chunks in any case and avoid re-downloading (and duplicate storage on disk), but this optimization hasn't been implemented as of Summer 2018.

Hyperdrive currently requires all files to be stored as contiguous chunks in the "content" hypercore feed. This is very performant, but makes de-duplication difficult. As a special case, it should be possible to support appending to the most recent file (which appends directly to the content feed) without copying the entire file. Any time any other file in the feed gets updated or created, that will break the contiguous chunk, but for your use case it might be good enough (if this optimization were to be implemented).