19

I'm looking for a way to create a virtual file system in Python for creating directories and files, before writing these directories and files to disk.

Using PyFilesystem I can construct a memory filesystem using the following:

>>> import fs
>>> dir = fs.open_fs('mem://')
>>> dir.makedirs('fruit')
SubFS(MemoryFS(), '/fruit')
>>> dir.makedirs('vegetables')
SubFS(MemoryFS(), '/vegetables')
>>> with dir.open('fruit/apple.txt', 'w') as apple: apple.write('braeburn')
... 
8
>>> dir.tree()
├── fruit
│   └── apple.txt
└── vegetables

Ideally, I want to be able to do something like:

dir.write_to_disk('<base path>')

To write this structure to disk, where <base path> is the parent directory in which this structure will be created.

As far as I can tell, PyFilesystem has no way of achieving this. Is there anything else I could use instead or would I have to implement this myself?

James Vickery
  • 732
  • 3
  • 10
  • 23

2 Answers2

21

You can use fs.copy.copy_fs() to copy from one filesystem to another, or fs.move.move_fs() to move the filesystem altogether.

Given that PyFilesystem also abstracts around the underlying system filesystem - OSFS - in fact, it's the default protocol, all you need is to copy your in-memory filesystem (MemoryFS) to it and, in effect, you'll have it written to the disk:

import fs
import fs.copy

mem_fs = fs.open_fs('mem://')
mem_fs.makedirs('fruit')
mem_fs.makedirs('vegetables')
with mem_fs.open('fruit/apple.txt', 'w') as apple:
    apple.write('braeburn')

# write to the CWD for testing...
with fs.open_fs(".") as os_fs:  # use a custom path if you want, i.e. osfs://<base_path>
    fs.copy.copy_fs(mem_fs, os_fs)
zwer
  • 24,943
  • 3
  • 48
  • 66
  • 1
    Woah I didn't expect something so simple! Thanks – James Vickery Jul 26 '18 at 23:02
  • Good answer. `copy_fs` accepts FS objects and FS URLs, so the following would also work: `fs.copy.copy_fs(mem_fs, '.')` – Will McGugan Aug 11 '18 at 16:06
  • 1
    So great! My use case is to populate an in-memory file system with objects and directories and only persist them to disk once my program has surpassed all failure legs and the output of the program is valid, otherwise, leave the previous files untouched. – John Carrell Apr 07 '20 at 16:22
3

If you just want to stage a file system tree in memory, look at the tarfile module.

Creating files and directories is a bit involved:

tarblob = io.BytesIO()
tar = tarfile.TarFile(mode="w", fileobj=tarblob)
dirinfo = tarfile.TarInfo("directory")
dirinfo.mode = 0o755
dirinfo.type = tarfile.DIRTYPE
tar.addfile(dirinfo, None)

filedata = io.BytesIO(b"Hello, world!\n")
fileinfo = tarfile.TarInfo("directory/file")
fileinfo.size = len(filedata.getbuffer())
tar.addfile(fileinfo, filedata)
tar.close()

But then you can create the file system hierarchy using TarFile.extractall:

tarblob.seek(0) # Rewind to the beginning of the buffer.
tar = tarfile.TarFile(mode="r", fileobj=tarblob)
tar.extractall()
Florian Weimer
  • 32,022
  • 3
  • 48
  • 92