10

The current architecture of my application doesn't allows me to store a file in server side and create the link to that stored file. So is there any other options (or a code snippet) to directly stream the ZipFile and store it at client side ?

Edit: I guess my question has been misinterpreted. I am receiving answers of Zipping the files and store it at client side. I have already achieve that. Below is the main concern with the sample use case:

Scenario: User has got around 5000 records (approx size of 1 MB each) and user want to download the child records( in CSV format) of each 5000 records compressed in ZIP format. All the CSV files are generated on the fly.

Approach: As the size of the ZIP file can be upto 5 GB, so I took the approach of direct streaming the content of files into the ZIP file created at the client side. I have used PipeInputStream and PipeOutputStream for this.

Result: As I am new to vaadin I am not successful in the above approach so looking for any suggestions/ code snippets which supports direct streaming of ZIP file(whatever the size may be) to the client side.

I guess I am now clear.

Milesh
  • 271
  • 1
  • 8
  • 24
  • 1
    On the link: https://vaadin.com/forum#!/thread/2824570 you can found discussion on this topic. – Dragan Radevic Apr 15 '16 at 11:52
  • Does [this](https://vaadin.com/docs/-/part/framework/application/application-resources.html#application.resources.stream) help in conjunction with [ZipOutputStream](http://www.mkyong.com/java/how-to-compress-files-in-zip-format/)? – Steffen Harbich Apr 15 '16 at 12:27
  • @DraganRadevic Thanks for sharing the link I already explored that link but the solution says that I need to store the file somewhere in my server and create the link so that user can download using that link. – Milesh Apr 18 '16 at 04:08
  • @SteffenHarbich nope. – Milesh Apr 18 '16 at 04:12
  • You can create ZipOutputStream in memory and then let user to download file. You have more options. To create BinaryInputStream (static content) or PipedInputStream if you want to download zip on the fly (while creating for large files) – Dragan Radevic Apr 18 '16 at 08:39
  • @DraganRadevic I have implemented using filedownloader of vaadin but in the case of large files it takes notable time. PipedInputStream sounds convincing though would you mind sharing some code snippets. Thanks – Milesh Apr 18 '16 at 09:23
  • Code examples: http://www.programcreek.com/java-api-examples/index.php?source_dir=cli-master/command/src/main/java/com/codenvy/cli/command/builtin/util/zip/ZipUtils.java https://gist.github.com/bobringer/4888cdedfcf026e1351d – Dragan Radevic Apr 18 '16 at 09:33
  • @DraganRadevic Thanks for sharing the link. Just wanted to make sure does the code supports direct streaming of ZipFile ( I have achieved my goal partially by creating the temp file and store it in client side ). – Milesh Apr 18 '16 at 09:56
  • @Milesh is it important for you to combine several files into 1 archive? – Tair May 04 '16 at 03:38
  • @tair as per the business requirement I have to create multiple CSV files, Zip into it and download the zip file. The whole activity should be covered in one session. One use case is there are 5000 CSV files and size of each file could be 1 MB and can increase when the feature goes into production. – Milesh May 04 '16 at 05:32
  • @Milesh do the CSV files exist beforehand or you also create them on the fly? I mean, do you need to take them from somewhere in file system, or you are taking the data from database? – Tair May 04 '16 at 05:52
  • @tair I also create them on the fly. – Milesh May 04 '16 at 06:24
  • @Milesh unfortunately, it is not easy to _efficiently_ transform a list of InputStreams into a ZippedInputStream. Java API only lets you do that using intermediate storage, that are not streamable out of box. That storage may be in-memory ByteArray (will eat your memory for big files) or temporary file (which you don't want to use). – Tair May 04 '16 at 06:34

5 Answers5

3

Your Question is not clear. Always, give some of your research while posting question. Now, Question is from where you want to stream some URL, Fileshare, Database. What Vaadin version you are using 6< or >6?

Edited There are two parts of problem.

  1. Downloading Zip file from website directly. Below is sample code snippet for downloading the any file.

    import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.URL;

import javax.servlet.annotation.WebServlet;

import com.vaadin.annotations.Theme;
import com.vaadin.annotations.VaadinServletConfiguration;
import com.vaadin.server.FileDownloader;
import com.vaadin.server.StreamResource;
import com.vaadin.server.StreamResource.StreamSource;
import com.vaadin.server.VaadinRequest;
import com.vaadin.server.VaadinServlet;
import com.vaadin.ui.Button;
import com.vaadin.ui.UI;

@SuppressWarnings("serial")
@Theme("vaadintest")
public class VaadintestUI extends UI {

    @WebServlet(value = "/*", asyncSupported = true)
    @VaadinServletConfiguration(productionMode = false, ui = VaadintestUI.class)
    public static class Servlet extends VaadinServlet {
    }

    protected void init(VaadinRequest request) {
        Button downloadButton = new Button("Download Zip File");

        StreamResource myResource = createResource();
        FileDownloader fileDownloader = new FileDownloader(myResource);
        fileDownloader.extend(downloadButton);

        setContent(downloadButton);
    }

private StreamResource createResource() {
    return new StreamResource(new StreamSource() {
        @Override
        public InputStream getStream() {

            BufferedInputStream in = null;
            ByteArrayOutputStream bao = null;
            try {
                in = new BufferedInputStream(
                        new URL("http://www-us.apache.org/dist/commons/io/binaries/commons-io-2.5-bin.zip" + "")
                                .openStream());
                byte[] buff = new byte[8000];
                int bytesRead = 0;
                bao = new ByteArrayOutputStream();

                while ((bytesRead = in.read(buff)) != -1) {
                    bao.write(buff, 0, bytesRead);
                }

            } catch (Exception e) {
                e.printStackTrace();
            }
            return new ByteArrayInputStream(bao.toByteArray());

        }
    }, "somefile.zip");
}

    }

  2. Zipping Huge CSVs files and download

Refer: Best Practices to Create and Download a huge ZIP (from several BLOBs) in a WebApp

Zipping and Storing in server side is easy. I don't think It will be easier to directly store to clients disk due to access issues (It is possible using OutputStream). You could zip it in server side and share link to client of zipped file(File need to be stored in WebApp folder).

Also, you could use output stream after zipping to download it.Below is simple example of it.

Refer: http://www.mkyong.com/java/how-to-download-file-from-website-java-jsp/

Edit Question, Give actual scenario: then it will be easy to share code snippet

Some Useful Links you can refer are below:

  1. Best Practices to Create and Download a huge ZIP (from several BLOBs) in a WebApp
  2. How to split a huge zip file into multiple volumes?
  3. Zip files with Java: Is there a limit?
  4. Very large zip file (> 50GB) --> ZipException: invalid CEN header
  5. Java multithreaded file downloading performance
  6. In Java: How to zip file from byte[] array?
  7. Uncompressing a ZIP file in memory in Java
  8. Best way to detect if a stream is zipped in Java
Community
  • 1
  • 1
Pratiyush Kumar Singh
  • 1,977
  • 3
  • 19
  • 39
  • Thanks for the response. I apologize for not sounding clear about my question. Please note that I have successfully created the csv files, zipped them and download them, BUT the concern is the size of the file which can be 5 GB. As per the business requirement I have to create multiple CSV files, Zip into it and download the zip file. The whole activity should be covered in one session. One use case is there are 5000 CSV files and size of each file could be 1 MB and can increase when the feature goes into production. Hope this helps. – Milesh May 04 '16 at 05:36
3

I can see that the answer has been accepted automatically by the system(NOT by me) as it was voted 3 times. I don't have an issue with that but I don't want to confuse the people who (in case) land up here searching for answer. I appreciate the answer given by @Pratyush Kumar Singh (as it helps you to explore more) but sorry StackOverFlow I can't accept the answer as my question hasn't been answered yet. I am still working on it and will post the answer once I resolved through POC.

Milesh
  • 271
  • 1
  • 8
  • 24
  • 1
    Hi @Milesh, after digging more around your issue I found out that your problem is that basically all the java ZIP implementations in the market only support push-style API (imperatively write to temporary OutputStream, then convert it to InputStream). Your problem can be solved with a pull-style API, in spirit of SequenceInputStream, but I couldn't find such implementation. I'm thinking of implementing it myself :) – Tair May 09 '16 at 08:57
  • @tair Exactly I agree with you. Please share with me once you found anything working :) – Milesh May 09 '16 at 10:20
  • hey @Milesh finally I got it! Look in my second answer! – Tair May 14 '16 at 09:51
2

You don't have to create a ZIP package to compress a single file over HTTP. All decent browsers support reading compressed HTTP response[1], so let's utilise that:

@Theme("mytheme")
@Widgetset("org.test.MyAppWidgetset")
public class MyUI extends UI {

    @Override
    protected void init(VaadinRequest vaadinRequest) {
        Button downloadButton = new Button("Download File");

        StreamResource myResource = createResource();
        FileDownloader fileDownloader = new FileDownloader(myResource);
        fileDownloader.extend(downloadButton);

        setContent(downloadButton);
    }

    private StreamResource createResource() {
        return new StreamResource(new StreamResource.StreamSource() {
            @Override
            public InputStream getStream() {
                InputStream s = null;
                try {
                    s = new DeflaterInputStream(new FileInputStream("/tmp/some.csv"));
                } catch (FileNotFoundException e) {
                    e.printStackTrace();
                }
                return s;

            }
        }, "some.csv") {
            @Override
            public DownloadStream getStream() {
                DownloadStream ds =  super.getStream();
                ds.setParameter("Content-Encoding", "deflate");
                return ds;
            }
        };
    }

    @WebServlet(urlPatterns = "/*", name = "MyUIServlet", asyncSupported = true)
    @VaadinServletConfiguration(ui = MyUI.class, productionMode = false)
    public static class MyUIServlet extends VaadinServlet {
    }
}

The browser will download the compressed stream, but save the original file, decompressing it on the fly.

Pros: this is very memory-efficient.

Cons: you can't combine several files into a single ZIP archive.

[1] https://en.wikipedia.org/wiki/HTTP_compression

Tair
  • 3,779
  • 2
  • 20
  • 33
  • Thanks for the response. But I guess above feature works when there is ONE file, here I have multiple files. – Milesh May 04 '16 at 05:38
2

The Java JDK and Apache commons-compress don't let you stream ZIP archives lazily, so I implemented a Java ZIP library [1] to handle that. The current limitation is it doesn't support ZIP64 extensions, it means it can't compress files bigger than 4 GiB and can't produce archives bigger than 4 GiB. I'm working on that.

[1] https://github.com/tsabirgaliev/zip

Tair
  • 3,779
  • 2
  • 20
  • 33
0

It looks like apache.commons.compress ZipArchiveOutputStream is the answer to your question. But you should know that there are a set of limitations for your situation (no RandomAccessFile). Simple sample ( use stdout redirect: java -cp Test > test.zip):

import org.apache.commons.compress.archivers.ArchiveEntry;
import org.apache.commons.compress.archivers.zip.ZipArchiveEntry;
import org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream;
import java.util.Arrays;

public class Test {

public static void main(String[] args) throws Exception {
      ZipArchiveOutputStream zipOut = new ZipArchiveOutputStream(System.out);
      byte[] data = new byte[1024*1024];
      for(byte i=49; i<120; i++) {
          Arrays.fill(data, i);
          ArchiveEntry file1 = new ZipArchiveEntry("file" + i + ".txt");
          zipOut.putArchiveEntry(file1);
          zipOut.write(data);
          zipOut.closeArchiveEntry();
       }
       zipOut.finish();
       zipOut.close();
    }
}
sibnick
  • 3,995
  • 20
  • 20