2

I'm new to Dagster.

According to the docs, when you use the built-in filesystem IO manager:

Subsequent materializations of an asset will overwrite previous materializations of that asset.

https://docs.dagster.io/_apidocs/io-managers#dagster.FilesystemIOManager

Indeed, if I change the code of an asset and re-materialize it, the new run gets a different code version and data version, however, the new result is written to the same path, overwriting the result of the previous materialization.

I find that strange, because in practice you usually need to keep track of the old data from previous runs, instead of overwriting it.

I think it's possible to create a custom IO manager that would include a unique run ID into the asset path. Is there a better, more out-of-the box way to preserve data from previous runs?

0 Answers0