2

I'm looking for the most memory efficient way to zip many large files using ColdFusion or Java. I have tried using <cfzip> and using zip.cfc by Nate Nielsen (http://farmancreative.com/womenskiteboarding/admin/dccom/components/dcFileManagerV3/actions/cfc/zip.cfc). For the tests I zipped up a directory that contains 80 mp4 files totaling 1.18GB. The results are below. I could not tell a difference at all when the <cfzip> tag was running, the normal "steps" of ColdFusion appeared unchanged. But with zip.cfc it was more "saw tooth" memory usage.

So my question is, which is the better result? Or is there another newer way that I don't know about that is better than both of these?

I care more about memory usage than speed. But as far as speed goes, <cfzip> was a little faster. <cfzip> time was 100,871. zip.cfc time was 141,285.

Thanks!enter image description here

<cfzip> Test Code:

<cfoutput>
    <cfset tBegin = GetTickCount()>
    <cfzip
        action="zip"
        source="#dir#"
        file="#zipFile#"
        storepath="false"
        overwrite="true"
        />
    <cfset tEnd = GetTickCount()>
    <cfset scriptTime = (tEnd - tBegin)>
    Script Time: #scriptTime#
</cfoutput>

zip.cfc Test Code:

<cfdirectory directory="#dir#" name="d" recurse="false">
<cfoutput>
    <cfset tBegin = GetTickCount()>
    <cfset zipper = createObject("component", "zip")>
    <cfscript>zipper.newZip(zipFile);</cfscript>
    <cfloop query="d">
        <cfset zipper.addFile(dir&d.name)>
    </cfloop>
    <cfscript>zipper.createZip();</cfscript>
    <cfset tEnd = GetTickCount()>
    <cfset scriptTime = (tEnd - tBegin)>
    Script Time: #scriptTime#
</cfoutput>
James A Mohler
  • 11,060
  • 15
  • 46
  • 72
gfrobenius
  • 3,987
  • 8
  • 34
  • 66
  • Uuh, compressing MPEG files? Do you actually gain that much space? – fge Mar 17 '14 at 21:11
  • No, none, was just testing large files and that's what I had. Would the file type make a difference? Both tests were ran against the same files. I'll gather up a slew of text files and pdfs and run tests against that and see what happens. I'm not testing compression here, mainly watching memory. – gfrobenius Mar 17 '14 at 21:13
  • Well, I cannot explain why, but yes, it makes a difference; attempting to compress "binary" files (for lack of a better term) takes longer than text files. If such files are not what you will be working with eventually, you should build yourself a representative set so that measurements be more accurate – fge Mar 17 '14 at 21:17
  • What fge says - test with the files that will be used! If you don't have enough of those files to perform a significant test, perhaps you don't need to be worrying about it? If it does matter, maybe consider something like [**7-zip**](http://www.7-zip.org/) - which gives you both more control/options and (in its GUI) provides estimated memory uses for compression/decompression based on options selected. – Peter Boughton Mar 17 '14 at 21:53
  • Thank you both. I am running tests as we speak with "real world" file types and the results are very different! Will post them shortly. Thanks. – gfrobenius Mar 17 '14 at 22:00
  • If you care about memory usage, try cfexecute and a command line zip tool. – Henry Mar 17 '14 at 22:18
  • @Henry we can only shell out like that for a couple things. Our environment is very locked down, to get winZip with command line support on the server would take a long approval process but we are considering it. – gfrobenius Mar 17 '14 at 22:21
  • 1
    Just actually looked at the zip.cfc code - none of the functions nor the component have `output=false` on them so it will be unnecessarily buffering (which will slow it down), and there is no var scoping (so it's not thread safe). Some parts of the code look rather dubious too - I would definitely **recommend _not_ using that zip.cfc** even if stuck on a version of CF before cfzip was added. – Peter Boughton Mar 17 '14 at 22:24
  • @PeterBoughton, thanks! Your comment on buffering peeked my interest. However I could not prove it as true. Please see this post I just created: **http://stackoverflow.com/questions/22486381/prove-that-coldfusion-functions-unnecessarily-buffer-with-output-false** – gfrobenius Mar 18 '14 at 17:14
  • 1
    Turn off whitespace management and view source. – Peter Boughton Mar 18 '14 at 17:32

1 Answers1

1

I have to run so can't type much right now but will come back to this tomorrow. Here are my test results after running it against real world file types (.txt, .ppt, .doc, .swf, etc...). Looks like <cfzip> is much better than zip.cfc.enter image description here

gfrobenius
  • 3,987
  • 8
  • 34
  • 66
  • I was encountering some performance issues w/CFZip w/Windows. I switched to using CFExecute with 7Zip instead of spending 30 seconds to 5 minutes using java. http://www.7-zip.org/ 7Zip is more robust (ie, support for more archive types, automatically deletion of source files upon completion). On the front end, I perform an ajax request to determine if the final ZIP file exists prior to forcing the download. Our application server has been much more stable as a result of doing this. – James Moberg Jul 13 '17 at 20:51