How to get the parent directory with fsspec?

Question

Given a filepath, how do I obtain the parent directory containing the file using fsspec ? Filepath can be using local filesystem or cloud storage, that's why fsspec is prefered.

Answer 1

fsspec implements file system operations, while paths are just strings. So, you don't need fsspec to manipulate paths. You can use Path.resolve or simply os.path.abspath, and these don't require access to the file system itself. For example:

>>> os.path.abspath("/foo/bar/..")
'/foo'

If you need to verify if the resolved parent path actually exists, this is where you'd use fsspec.spec.AbstractFileSystem.exists which will actually make a request to the underlying file system.

See also this question: python : how to get absolute path for a parent dir

Answer 2

@Timur is correct in paths just being strings, and you are welcome to manipulate them directly.

However, any filesystem also implements a _parent class method, which will give you a normalised version of the parent directory (normalised meaning stripping the protocol and host, converting windows to posix, etc, and can in principle be backend-dependent).

fs = fsspec.filesystem("s3")
parent = fs._parent("s3://bucket/prefix/path")
assert parent == "bucket/prefix"

We also intend to implement a filesystem .path attribute to provide convenience os.path-like functions specialised to the filesystem in question.