3

In the API docs about kedro.io and kedro.contrib.io I could not find info about how to read/write data from/to network attached storage such as e.g. FritzBox NAS.

thinwybk
  • 4,193
  • 2
  • 40
  • 76

1 Answers1

3

So I'm a little rusty on network attached storage, but:

  1. If you can mount your network attached storage onto your OS and access it like a regular folder, then it's just a matter of providing the right filepath when writing the config for a given catalog entry. See for example: Using Python, how can I access a shared folder on windows network?

  2. Otherwise, if accessing the network attached storage requires anything special, you might want to create a custom dataset that uses a Python library for interfacing with your network attached storage. Something like pysmb comes to mind.

The custom dataset could borrow heavily from the logic in existing kedro.io or kedro.extras.datasets datasets, but you replace the filepath/fsspec handling code with pysmb instead.

Zain Patel
  • 983
  • 5
  • 14
  • Thanks. This helps a lot. On a linux machine one can auto mount into the local file system as well. https://github.com/Pantastisch/fritz_nas_auto_mount – thinwybk May 14 '20 at 09:49
  • Perfect. Looks like you can use any dataset and just provide `/mnt/fritz_nas/` as the file path (for e.g). By the way, I'm curios -- what's the use case for putting datasets on a NAS? – Zain Patel May 14 '20 at 10:01
  • Pls do not ask. :smile: The team wants to share raw data in the local network. Usually I'd setup a Minio based, S3-like blob storage in the local network. The kendro.io S3 functionality should work with it as is. However it's some constraint I've to work with. – thinwybk May 14 '20 at 10:18
  • 2
    I've created a class implementing just as much basic functionality as I need. Feel free to create a plugin from the source code: https://gist.github.com/fkromer/42eb93cbb8e4a58e57fd4734e208e568 – thinwybk Jun 03 '20 at 09:11