How can I modify a zip file in memory only?

Question

I have a Ruby application, and I need to modify an existing zip file.

I want to build the zip file in memory and stream back the bytes without ever writing the file to the filesystem. If I end up hosting this on Heroku I don't think I can write to the filesystem. Does anyone know of a way to do that?

I looked at Zip::ZipFile but it looks like it always wants to write to the filesystem. I figured being "based on the java implementation" I would be able to just get the compressed file's bytes, which you can do in java, but I don't see a way to do that.

Edit:

What I am asking is basically the same as this, but for Ruby instead of Python: Function to create in-memory zip file and return as http response

score 4 · Answer 1 · edited Jul 06 '21 at 09:50

had same issue, got to get it work by closing the file and reading the data and streaming it as send_data

then found another library that works fine on heroku and can handle with in-memory buffers: it's zipruby (not rubyzip).

buffer = ''
Zip::Archive.open_buffer(buffer, Zip::CREATE) do |archive|
  files.each do |wood, report|
    title = wood.abbreviation+".txt"
    archive.add_buffer(title, report);
  end
end
file_name = "dimter_#{@offer.customerName}_#{Time.now.strftime("%m%d%Y_%H%M")}.zip"
send_data buffer, :type => 'application/zip', :disposition => 'attachment', :filename => file_name

score 3 · Answer 2 · edited May 23 '17 at 10:26

3

Here's a blog post which deals with this issue. It uses Tempfile and seems like an excellent solution to me (though read through the comments for some useful additional discussion).

An example, from the post:

def download_zip(image_list)
  if !image_list.blank?
    file_name = "pictures.zip"
    t = Tempfile.new("my-temp-filename-#{Time.now}")
    Zip::ZipOutputStream.open(t.path) do |z|
      image_list.each do |img|
        title = img.title
        title += ".jpg" unless title.end_with?(".jpg")
        z.put_next_entry(title)
        z.print IO.read(img.path)
      end
    end
    send_file t.path, :type => 'application/zip',
                      :disposition => 'attachment',
                      :filename => file_name
    t.close
  end
end

This solution should play nice with Heroku.

edited May 23 '17 at 10:26

Community

1
1

answered Dec 03 '10 at 23:01

Jordan Running

102,619
17
182
182

Doesn't Tempfile create a file? – Mark Thomas Dec 03 '10 at 23:31
1

Yes it does, Mark (unless your temp directory lives in memory), but rally25rs didn't specify why he didn't want to create a file. I'm making an assumption that led me to a solution that a) will work fine on Heroku, and b) creates a file, but one rally25rs will never have to think about again, and which will get cleaned up by the OS. If this doesn't solve his/her core problem I'd like to know. – Jordan Running Dec 03 '10 at 23:57
I guess my original intent was to work around this Heroku constraint of a read-only filesytem: http://docs.heroku.com/constraints#read-only-filesystem but I guess I can write to temp file to the /tmp directory if needed. I'm pretty new to Ruby, but implementing this in Java would be very simple to just keep the entire file in a memory buffer, so I figured it would have been in Ruby too. Thanks for the help! – CodingWithSpike Dec 04 '10 at 18:35

score 1 · Answer 3 · answered Dec 04 '10 at 02:17

1

You could always patch the new and open methods of Zip::ZipFile to allow use of StringIO handles, then do your I/O directly to memory.

answered Dec 04 '10 at 02:17

the Tin Man

158,662
42
215
303

CodingWithSpike · Answer 4 · 2010-12-05T17:06:31.833

Going to propose an answer to my own question here, that I think better fits what I was trying to do. This method really makes no file (no temp file).

Since ZipFile extends, and is really just a bunch of convenience methods around ZipCentralDirectory, you can work directly with ZipCentralDirectory instead of ZipFile. That will alow you to use IO streams for creating and writing a zip file. Plus throw in use of StringIO and you can do it from a string:

  # load a zip file from a URL into a string
  resp = Net::HTTP.new("www.somewhere.com", 80).get("/some.zip")
  zip_as_string = response.body

  # open as a zip
  zip = Zip::ZipCentralDirectory.read_from_stream(StringIO.new(zip_as_string))

  # work with the zip file.
  # i just output the names of each entry to show that it was read correctly
  zip.each { |zf| puts zf.name }

  # write zip back to an output stream
  out = StringIO.new
  zip.write_to_stream(out)

  # use 'out' or 'out.string' to do whatever with the resulting zip file.
  out.string

Update:

This actually doesn't work at all. It will write a readable zip file, but ONLY the zip file's 'table of contents'. All the internal files are 0 length. Digging further into the Zip implementation, it looks like it only holds the zip entry 'metadata' in memory, and it goes back to the underlying file to read everything else. Based on this, it looks like it is impossible to use the Zip implementation at all without writing to the filesystem.

How can I modify a zip file in memory only?

4 Answers4

Linked