0

One of pyfilesystem's main feature is virtual filesystems. E.g.

home_fs = open_fs('~/')
projects_fs = home_fs.opendir('/projects')

I think that is a great feature and was hoping that fsspec has something similar. But I couldn't find an example and I'm not able to get it working.

Juergen
  • 699
  • 7
  • 20

1 Answers1

0

You might want DirFileSystem, invoked like

fs = fsspec.implementations.dirfs.DirFileSystem(
    "<root path>", fs=fsspec.filesystem("file")
)

You can apply this to any filesystem, not only local. root_path needs to be a string that, when you affix further path parts to it, makes a complete path the target filesystem can understand; it may include the protocol (e.g., for HTTP, it must do). In your case, it would be "~" (or the expanded version of this, which would be more explicit).

Alternatively, you can create an arbitrarily mapped virtual filesystem with ReferenceFileSystem.

mapping = {"/key1": ["/local/path/file1"],
           "/key2": ["/other/unrelated/path/file"]}
fs = fsspec.filesystem("reference", fo=mapping)

Here, fs.cat("/key1") would get the contents of "/local/path/file1". You can have those paths be remote, or a mix of different backends, and even byte ranges of target files.

mdurant
  • 27,272
  • 5
  • 45
  • 74
  • Regarding the DirFileSystem example, and after reading the docs, I don't yet understand the 'fs' parameter. Does it need to match the protocol in ""? Is it always "file"? Or is "" something special, like the path without any protocol? – Juergen Nov 11 '22 at 08:00
  • Edited with what I hope is clarification. You are welcome to propose improvements to the class documentation. – mdurant Nov 11 '22 at 14:16