Examples that send unkown size data with http chunked header?

Question

I still don't have a clear picture of practical examples of the chunked header usage, after reading some posts and Wikipedia.

One example I see from Content-Length header versus chunked encoding, is:

On the other hand, if the content length is really unpredictable beforehand (e.g. when your intent is to zip several files together and send it as one), then sending it in chunks may be faster than buffering it in server's memory or writing to local disk file system first.

So it means that I can send zip files while I am zipping them ? How ?

I've also noticed that if I download a GitHub repo, I am receiving data in chunked. Does GitHub also send files in this way (sending while zipping) ?

A minimal example would be much appreciated. :)

*”I can send zip files while I am zipping them ? How ?”* — A program that produces a compressed file just outputs a *stream of bytes*. If you write that to a file on disk, you have a ZIP file. But you can also just output that stream over HTTP on the fly. How to do that exactly depends on your programming language. — deceze, Jun 08 '19 at 05:06
@deceze Hmm... I get the point. But I do want to see a real code example :0. Python C++ php exmaples are ok. Any material recommended ? — Rick, Jun 08 '19 at 05:17

Mobrine Hayde · Answer 1 · 2019-06-08T10:09:16.157

Here is an example using perl (with IO::Compress::Zip module) to send a zipped file on the fly as @deceze pointed to

use IO::Compress::Zip qw(:all);

my @files = ('example.gif', 'example1.png'); # here are some files

my $path = "/home/projects/"; # files location

# here is the header
print "Content-Type: application/zip\n"; # we are going to compress to zip and send it
print "Content-Disposition: attachment; filename=\"zip.zip\"\r\n\r\n"; # zip.zip for example is where we are going to zip data

my $zip = new IO::Compress::Zip;

foreach my $file (@files) {
  $zip->newStream(Name => $file, Method => ZIP_CM_STORE); # storing files in zip
  open(FILE, "<", "$path/$file");
  binmode FILE; # reading file in binary mode

  my ($buffer, $data, $n);

  while (($n = read FILE,$data, 1024) != 0) { # reading data from file to the end
     $zip->print($data); # print the data in binary
  }

  close(FILE);
}
$zip->close;

As you see in the script so even if you add the zip filename in the header, it doesn't matter, because we are zipping the files and printing it in binary mode right away, so it's not necessary to zip the data and store them then send it to the client, you can directly zip the files and print them without storing it.

Examples that send unkown size data with http chunked header?

1 Answers1

Linked