3

I have a lot of images in my python memory that I would like to place in an in memory tar, after which I would like to flush that tar to the disk.

I would like to do this in memory to prevent the writing of a lot of tiny files to my disk.

What I have so far is a way to save the images as memory files

import imageio

im = np.zeros((256,256))
out_file =  io.BytesIO()
imageio.imsave(out_file, im, format = 'jpg')

I can also place these in memory files into an in memory tar.

import tarfile

t = tarfile.TarInfo("helloworld.tif")
t.size = len(out_file.getbuffer())
tarBuffer = io.BytesIO()
tar =  tarfile.TarFile(mode="w", fileobj=tarBuffer)
tar.addfile(t, io.BytesIO(out_file.getbuffer()))

But now I do not know how to in turn get this in memory tar file to my disk.

Does anyone have any suggestions how to do this?

Daan
  • 349
  • 4
  • 16
  • Seems like you'd so something like in [this](https://stackoverflow.com/a/18457698/6273251) post; just open a file handle `f` to the path of the eventual tarfile you want to write to disk, and then do something like `f.write(tarBuffer.getvalue())` after `seek`ing to the beginning. – Random Davis Dec 21 '21 at 23:55

1 Answers1

0

Ok after quite some twiggeling I finally got something to work.

import numpy as np
import io
import tarfile
import imageio
import time

#create dummy image
im = np.zeros((256,256))

#create memory file
out_file =  io.BytesIO()
imageio.imsave(out_file, im, format = 'jpg')

#create memory tar
tarBuffer = io.BytesIO()

#write memory file to memory tar
t = tarfile.TarInfo("helloworld.tif")
t.size = len(out_file.getbuffer())
with tarfile.open("foo.tar", mode="w:gz", fileobj= tarBuffer) as tar:
    tar.addfile(t, io.BytesIO(out_file.getbuffer()))

#write memory tar to disk
tarBuffer.seek(0, 0)
with open('out.tar', 'wb') as dump:
    dump.write(tarBuffer.read()) 

#close the files
out_file.close()
tarBuffer.close()
Daan
  • 349
  • 4
  • 16