Unable to process large files in camel using http4

Question

A scenario, need to consume a rest webservice which provides a huge file as a stream output and vice versa need to handle the stream and directly write to a file rather memory. Service :

@RequestMapping(value = "downloadFile", method = RequestMethod.GET)
public StreamingResponseBody getSteamingFile(HttpServletResponse response) throws IOException {
    response.setContentType("application/octet-stream");
    InputStream inputStream = new FileInputStream(new File("data\\test_big.txt"));
    return outputStream -> {
        int nRead;
        byte[] data = new byte[1024];
        System.out.println("Writing some bytes..");
        while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
            outputStream.write(data, 0, nRead);
        }
        System.out.println("Completed #####");  
        outputStream.flush();
        outputStream.close();
        response.flushBuffer();
    };

}

Consumer Route:

.to("http4://localhost:8080/downloadFile")
        .process(new Processor() {
            public void process(Exchange exchange) throws Exception {
                InputStream is = exchange.getIn().getBody(InputStream.class);
                File ret = File.createTempFile("loadTest", "tmp");
                FileOutputStream fos = new FileOutputStream(ret);
                StreamUtils.copy(is, fos);
                System.out.println("File Name "+ ret.getName());
                is.close();
                fos.flush();
                fos.close();
            }
        });

256 JVM is going out of Memory when processing 300 MB which treats my route is not performing streaming to file.

score 0 · Answer 1 · answered Apr 15 '20 at 13:33

.to("http4://localhost:8080/downloadFile?disableStreamCache=true")

disableStreamCache: Determines whether or not the raw input stream from Servlet is cached or not (Camel will read the stream into a in memory/overflow to file, Stream caching) cache. By default Camel will cache the Servlet input stream to support reading it multiple times to ensure it Camel can retrieve all data from the stream. However you can set this option to true when you for example need to access the raw stream, such as streaming it directly to a file or other persistent store

score -1 · Answer 2 · answered Apr 12 '20 at 15:11

-1

You have to enable stream caching read here.

answered Apr 12 '20 at 15:11

S Shukla

43
8

This would only help if you stream large payloads to disk. Which introduces a pretty hefty performance penalty. In principle it should be possible to process everything as streams and with that avoid having to load the payload into memory (or buffer on disk). – Ralf Apr 12 '20 at 20:50
If you see consumer code, it is already dumping it to temp file. – S Shukla Apr 13 '20 at 05:48
I assume this is only to demonstrate the problem. The file has the base name `loadTest` after all? – Ralf Apr 13 '20 at 07:07
Thanks for all your time in giving replies, finally i got solution by enabling a feature in http4. --- http4://localhost:8061?disableStreamCache=true – Sudha Mokshagundam Apr 15 '20 at 10:34
I Tried with 64 MB RAM and consumed web service and able to write 143 MB file using above disbaleStreamCache property which performed well and potentially there is a crest in CPU as its purley I/O. – Sudha Mokshagundam Apr 15 '20 at 10:37

Unable to process large files in camel using http4

2 Answers2