6

I am wanting to download a tar.gz archive, extract it and compress it into a zip file in one command with a bash script. Reason for this is to be independent from temporary files.

The code I use:

curl -L "someURL" | tar xOz --strip-components=1 | zip -@ test.zip

gives a lot of output to STDOUT so I guess zip is not accepting the pipe.

Maybe I am missing something out here, but the man page of zip doesn't give me more information than using -@ or - nor does the internet.

Flatron
  • 1,375
  • 2
  • 12
  • 33
  • 1
    May be [this thread][1] can help you. [1]: http://stackoverflow.com/questions/2019603/how-do-you-specify-filenames-within-a-zip-when-creating-it-on-the-command-line-fr – Stefano Falsetto Sep 15 '14 at 15:40

2 Answers2

3

The manpage for zip says (at least on my system):

If a file list is specified as -@ [Not on MacOS], zip takes the list of input files from standard input instead of from the command line. For example,
zip -@ foo
will store the files listed one per line on stdin in foo.zip.

The manpage for tar

-O, --to-stdout
             extract files to standard output.

So, in short:

tar -O can output the files (but not their names) in one long stream to stdout. But zip expects a list of filenames on stdin. So that's not going to work. And it's hard to see how to make it work, because bash pipes are just unstructured strings, but to transmit the information from tar to zip you need to add some structure, even if it is minimal:

[filename][filedata][filename][filedata]...

And the sender (tar) and receiver (zip) would have to agree on the format of that structure. Which is not going to happen.

However, you can use interfaces to tar and zip other than the command-line utilities. For example, if you have python installed, the following should work:

#!/usr/bin/python
import sys
import tarfile
import zipfile
tarf = tarfile.open(sys.argv[1], "r:*")
zipf = zipfile.ZipFile(sys.argv[2], "w", zipfile.ZIP_DEFLATED)
for m in tarf:
  if m.isreg():
    zipf.writestr(m.path, tarf.extractfile(m).read())

(Needs lots of error checking. As written, it just crashes on any error.)

You can make that into a shell "one-very-long-liner" although personally I'd just use the python script above.

 python -c "$(printf %s \
   'import sys;import tarfile;import zipfile;' \
   'T=tarfile.open(sys.argv[1],"r:*")' \
   'Z=zipfile.ZipFile(sys.argv[2],"w",zipfile.ZIP_DEFLATED);' \
   '[Z.writestr(m.path,T.extractfile(m).read()) for m in T if m.isreg()]')" \
   input.tar output.zip

(If you want to pipe from curl into that, use /dev/stdin as the input file. I think that will avoid Python trying to interpret stdin as a UTF-8 stream.)

rici
  • 234,347
  • 28
  • 237
  • 341
  • `tar` can output *either* the contents or their names to `stdout` but not both. Nor would it being able to do that be all that useful unless something else understood its output format for that. – Etan Reisner Sep 15 '14 at 15:51
  • @EtanReisner: Yeah, the point (which you also make) is that bash pipes are unformatted strings and unformatted strings suck as a mechanism for serializing structured data. So you need to use something a bit more sophisticated. I did it in python, but other bindings are possible. – rici Sep 15 '14 at 16:38
  • Add "allowZip64 = True" parameter to "ZipFile" to support files greater than 4 GB. – Skarllot Aug 08 '18 at 03:38
2

tar is going to send all the file data to stdout (but no file names).

zip can't possibly do much of anything sane with that (barring creating a giant zip blob of doom of all the file contents in a single zip file and I can't imagine you want that).

You need to extract the files to disk if you want to create a zip archive of them.

I was going to say that you might be able to loop over the entries in the tarball (by name) and extract each one to the pipe (though that would be very costly in terms of number of times needed to scan through the tarball) but I don't actually see, in the man page for zip I have here at least, a way to get zip to compress data given to it via standard input. It only seems to take file names that way.

Etan Reisner
  • 77,877
  • 8
  • 106
  • 148
  • Thank you for this explanation. So what you are about to say is, that there is no way today to zip from stdout without a file list. So this leaves me at the beginning again, trying to find a workaround. – Flatron Sep 15 '14 at 15:46
  • @Flatron From what the `zip` manpage says (as quoted by @rici in his answer) no. It cannot take file contents as input only a list of filenames. Recall that `zip` doesn't come from the same sort of universe as `tar` and friends do. – Etan Reisner Sep 15 '14 at 15:50