Best way to Pipe InputStream to OutputStream

Question

I was to trying to find the best way to pipe the InputStream to OutputStream. I don't have an option to use any other libraries like Apache IO. Here is the snippet and output.

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.channels.FileChannel;

public class Pipe {
    public static void main(String[] args) throws Exception {

        for(PipeTestCase testCase : testCases) {
            System.out.println(testCase.getApproach());
            InputStream is = new FileInputStream("D:\\in\\lft_.txt");
            OutputStream os = new FileOutputStream("D:\\in\\out.txt");

            long start = System.currentTimeMillis();            
            testCase.pipe(is, os);
            long end = System.currentTimeMillis();

            System.out.println("Execution Time = " + (end - start) + " millis");
            System.out.println("============================================");

            is.close();
            os.close();
        }

    }

    private static PipeTestCase[] testCases = {

        new PipeTestCase("Fixed Buffer Read") {         
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                byte[] buffer = new byte[1024];
                while(is.read(buffer) > -1) {
                    os.write(buffer);   
                }
            }
        },

        new PipeTestCase("dynamic Buffer Read") {           
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                byte[] buffer = new byte[is.available()];
                while(is.read(buffer) > -1) {
                    os.write(buffer);   
                    buffer = new byte[is.available() + 1];
                }
            }
        },

        new PipeTestCase("Byte Read") {         
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                int c; 
                while((c = is.read()) > -1) {
                    os.write(c);    
                }
            }
        }, 

        new PipeTestCase("NIO Read") {          
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                FileChannel source      = ((FileInputStream) is).getChannel(); 
                FileChannel destnation  = ((FileOutputStream) os).getChannel();
                destnation.transferFrom(source, 0, source.size());
            }
        }, 

    };
}


abstract class PipeTestCase {
    private String approach; 
    public PipeTestCase( final String approach) {
        this.approach = approach;           
    }

    public String getApproach() {
        return approach;
    }

    public abstract void pipe(InputStream is, OutputStream os) throws IOException;
}

Output (~4MB input file) :

Fixed Buffer Read
Execution Time = 71 millis
============================================
dynamic Buffer Read
Execution Time = 167 millis
============================================
Byte Read
Execution Time = 29124 millis
============================================
NIO Read
Execution Time = 125 millis
============================================

'Dynamic Buffer Read' uses available() method. But it is not reliable as per java docs

It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.

'Byte Read' seems to be very slow.

So 'Fixed Buffer Read' is the best option for pipe? Any thoughts?

did you mean the apache IO is better – shareef Apr 02 '15 at 15:01 — shareef, Apr 02 '15 at 15:01

score 20 · Answer 1 · edited Nov 16 '22 at 10:29

20

Java 9

Since Java 9 one can use this method from InputStream:

public long transferTo(OutputStream out) throws IOException

Pre Java 9

A one-liner from apache commons:

IOUtils.copy(inputStream, outputStream);

Documentation here. There are multiple copy methods with different parameters. It is also possible to specify the buffer size.

edited Nov 16 '22 at 10:29

gouessej

3,640
3
33
67

answered Jan 12 '16 at 11:15

Dariusz

21,561
9
74
114

1

This doesn't answer the question. The question said the answer can't use other libraries. – Nathan Feb 03 '16 at 17:23
2

you're right. he didn't answer the question. But, the answer did give me a solution since I could use 3rd party. I would edit the answer to acknowledge the fact it doesn't answer his need, but it's a useful answer for others like me. So my thanks. – Perry Tew May 27 '16 at 20:00
Source library is written explicitely at the start. Google indexes titles best, hence this question has a pretty high view count. The "other libraries" remark is not as highly indexed. – Dariusz May 28 '16 at 09:50
1

It may not answer @OP's needs, but it answers mine, and I can't very well ask a new question can I? It would just be linked to this question and then I'd be told to not ask questions that has already been answered. Since search engines send us here, there's even more of a reason to have these extra answers: They help those who aren't OP, but have almost the same problem. – Haakon Løtveit Aug 26 '19 at 11:35

score 14 · Answer 2 · answered Jan 31 '13 at 20:50

14

I came across this, and the final read can cause problems.

SUGGESTED CHANGE:

public void pipe(InputStream is, OutputStream os) throws IOException {
  int n;
  byte[] buffer = new byte[1024];
  while((n = is.read(buffer)) > -1) {
    os.write(buffer, 0, n);   // Don't allow any extra bytes to creep in, final write
  }
 os.close ();

I also agree that 16384 is probably a better fixed buffer size than 1024.

IMHO...

answered Jan 31 '13 at 20:50

paulsm4

114,292
17
138
190

1

Buffer size should be a parameter then, different use cases call for different buffer sizes. – Mmmh mmh Aug 20 '14 at 13:31
@AurélienOoms No they don't. The code in this answer will work for any buffer size greater than zero. – user207421 Feb 16 '17 at 10:30

score 12 · Accepted Answer · answered Jul 30 '12 at 06:00

12

I would say a fixed buffer size is the best/easiest to understand. However there are a few problems.

You're writing the entire buffer to the output stream each time. For the final block the read may have read < 1024 bytes so you need to take this into account when doing the write (basically only write number of bytes returned by read()
In the dynamic buffer case you use available(). This is not a terribly reliable API call. I'm not sure in this case inside a loop whether it will be ok, but I wouldn't be suprised if it was implemented sub-optimally in some implementations of InputStream.
The last case you are casting to FileInputStream. If you intend for this to be general purpose then you can't use this approach.

answered Jul 30 '12 at 06:00

Mike Q

22,839
20
87
129

Agree with all three points. The fixed buffer case needs to fixed. The dynamic buffer MHO is just broken - I wouldn't be surprised if a situation occurs where `available()` returns 0 before the end is reached (i.e. when reading from network connection). – Axel Jul 30 '12 at 06:45
@Mike Q, NIO is not option for me right now. the interface defines the InputStream and OutputStream. – Siva Arunachalam Jul 30 '12 at 08:01
I normally use fixed buffer size at 16k or more, as most people have a spare 16k of ram these days :) Also, if possible it's good to know how many bytes are coming in so you don’t end the writing too soon. – complistic Jan 23 '13 at 11:14

score 8 · Answer 4 · edited Mar 16 '16 at 09:03

8

java.io contains PipedInputStream and PipedOutputStream

PipedInputStream input = new PipedInputStream();
PipedOutputStream output = new PipedOutputStream (input);

write to input and it will be visible in output as an Outputstream. Things can work the other way around as well

edited Mar 16 '16 at 09:03

Chirlo

5,989
1
29
45

answered Feb 16 '16 at 12:18

Radu Simionescu

4,518
1
35
34

2

The OP already has both input and output - how do you connect both ends with these piped streams? – OrangeDog Aug 01 '16 at 17:10
the op question is not really about pipes... it's more about how to copy one file to another. A pipe is more about passing an output stream as the input stream for another process. – Radu Simionescu Aug 01 '16 at 19:48
"write to input" - did you mean write to output? – Llew Vallis Aug 25 '18 at 09:04

Best way to Pipe InputStream to OutputStream

4 Answers4

Linked

Related