22

I tried to copy an InputStream to a File, and abort the copy if the size of InputStream is greater than 1MB. In Java7, I wrote code as below:

public void copy(InputStream input, Path target) {
    OutputStream out = Files.newOutputStream(target,
            StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
    boolean isExceed = false;
    try {
        long nread = 0L;
        byte[] buf = new byte[BUFFER_SIZE];
        int n;
        while ((n = input.read(buf)) > 0) {
            out.write(buf, 0, n);
            nread += n;
            if (nread > 1024 * 1024) {// Exceed 1 MB
                isExceed = true;
                break;
            }
        }
    } catch (IOException ex) {
        throw ex;
    } finally {
        out.close();
        if (isExceed) {// Abort the copy
            Files.deleteIfExists(target);
            throw new IllegalArgumentException();
        }
    }}
  • First question: is there any better solution for it?
  • Second question: my other solution - Before the copy operation, I calculate the size of this InputStream. So I copy the InputStream to ByteArrayOutputStream then get size(). But the problem is InputStream may not markSupported(), so the InputStream cannot be reused in a copy file operation.
Arjan Tijms
  • 37,782
  • 12
  • 108
  • 140
卢声远 Shengyuan Lu
  • 31,208
  • 22
  • 85
  • 130
  • I would do the test before the write, not after it. You can't 'calculate the size' of an `InputStream`. It could be infinite. The concept has no meaning. Playing tricks with `mark()` and `reset()` can only work if you download the entire stream, which is exactly what you're trying to avoid. – user207421 Jul 21 '16 at 01:57

6 Answers6

23

My personal choice is an InputStream wrapper that counts bytes as it reads them:

public class LimitedSizeInputStream extends InputStream {

    private final InputStream original;
    private final long maxSize;
    private long total;

    public LimitedSizeInputStream(InputStream original, long maxSize) {
        this.original = original;
        this.maxSize = maxSize;
    }

    @Override
    public int read() throws IOException {
        int i = original.read();
        if (i>=0) incrementCounter(1);
        return i;
    }

    @Override
    public int read(byte b[]) throws IOException {
        return read(b, 0, b.length);
    }

    @Override
    public int read(byte b[], int off, int len) throws IOException {
        int i = original.read(b, off, len);
        if (i>=0) incrementCounter(i);
        return i;
    }

    private void incrementCounter(int size) throws IOException {
        total += size;
        if (total>maxSize) throw new IOException("InputStream exceeded maximum size in bytes.");
    }

}

I like this approach because it is transparent, it is re-usable with all input streams and it works well with other libraries. For example copying files up to 4KB with Apache Commons:

InputStream in = new LimitedSizeInputStream(new FileInputStream("from.txt"), 4096);
OutputStream out = new FileOutputStream("to.txt");
IOUtils.copy(in, out);

PS: The main difference of the implementation above with BoundedInputStream is that BoundedInputStream does not throw an exception when the limit is exceeded (it just closes the stream)

idrosid
  • 7,983
  • 5
  • 44
  • 41
  • 2
    For simple file upload cases : `if (IOUtils.copy(new BoundedInputStream(inputStream, MAX_SIZE + 1), new FileOutputStream(tmpFile)) > MAX_SIZE) { /* error */}` – btpka3 Mar 23 '18 at 03:11
  • This approach makes sense but why do you incrementCounter(1) by 1 in the read() method? Shouldn't it be incremented by i? Unless original.read() is always returning 1? I'm not sure it is possible to assume that. Also, it seems that you're missing the overrides for close(), otherwise your wrapped stream might never be disposed. Thanks for your answer. – Ruben Daddario Jul 28 '20 at 18:12
  • 1
    Consider extending `FilterInputStream` instead of `InputStream` as it is the recommended way to implement de decorator of type `InputStream` – Nicolas Filotto May 28 '21 at 16:15
17

There are following ready solutions for this:

Matt Sheppard
  • 116,545
  • 46
  • 111
  • 131
husayt
  • 14,553
  • 8
  • 53
  • 81
  • 4
    ```uploadFile(ByteStreams.limit(stream, maxSize)); // TODO How would you figure out whether or not you hit the maxSize limit? ``` – RvPr Jan 05 '18 at 17:31
  • Great, exactly what I was looking for! Why invent the wheel again? – JohnyTex Oct 11 '18 at 14:54
3

First question: is there any better solution for it?

Not really. Certainly, not significantly better.

Second question: my other solution - Before the copy operation, I calculate size of InputStream. So I copy the InputStream to ByteArrayOutputStream then get size(). But the problem is InputStream may not markSupported(), so the InputStream cannot be reused in copy file operation.

Leaving aside that the above is a statement not a question ...

If you have copied the bytes to the ByteArrayOutputStream, you can then create a ByteArrayInputStream from the byte array returned by baos.toByteArray(). So you don't need to mark / reset the original stream.

However, that is a pretty ugly way of implementing this. Not least because you are reading and buffering the entire input stream anyway.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • Thanks! Do you mean you agree with my `copy()` solution? – 卢声远 Shengyuan Lu Mar 16 '13 at 04:36
  • 2
    Correct. If MOST of the calls to the method resulted in aborts, then you might consider reading up to 1Mb into a buffer, and only creating the output file if the input isn't too big. But I doubt that that is going to be true. – Stephen C Mar 16 '13 at 04:36
2

This is the implementation from Apache Tomcat:

package org.apache.tomcat.util.http.fileupload.util;

import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;

/**
 * An input stream, which limits its data size. This stream is
 * used, if the content length is unknown.
 */
public abstract class LimitedInputStream extends FilterInputStream implements Closeable {

    /**
     * The maximum size of an item, in bytes.
     */
    private final long sizeMax;

    /**
     * The current number of bytes.
     */
    private long count;

    /**
     * Whether this stream is already closed.
     */
    private boolean closed;

    /**
     * Creates a new instance.
     *
     * @param inputStream The input stream, which shall be limited.
     * @param pSizeMax The limit; no more than this number of bytes
     *   shall be returned by the source stream.
     */
    public LimitedInputStream(InputStream inputStream, long pSizeMax) {
        super(inputStream);
        sizeMax = pSizeMax;
    }

    /**
     * Called to indicate, that the input streams limit has
     * been exceeded.
     *
     * @param pSizeMax The input streams limit, in bytes.
     * @param pCount The actual number of bytes.
     * @throws IOException The called method is expected
     *   to raise an IOException.
     */
    protected abstract void raiseError(long pSizeMax, long pCount)
            throws IOException;

    /**
     * Called to check, whether the input streams
     * limit is reached.
     *
     * @throws IOException The given limit is exceeded.
     */
    private void checkLimit() throws IOException {
        if (count > sizeMax) {
            raiseError(sizeMax, count);
        }
    }

    /**
     * Reads the next byte of data from this input stream. The value
     * byte is returned as an <code>int</code> in the range
     * <code>0</code> to <code>255</code>. If no byte is available
     * because the end of the stream has been reached, the value
     * <code>-1</code> is returned. This method blocks until input data
     * is available, the end of the stream is detected, or an exception
     * is thrown.
     * <p>
     * This method
     * simply performs <code>in.read()</code> and returns the result.
     *
     * @return     the next byte of data, or <code>-1</code> if the end of the
     *             stream is reached.
     * @throws  IOException  if an I/O error occurs.
     * @see        java.io.FilterInputStream#in
     */
    @Override
    public int read() throws IOException {
        int res = super.read();
        if (res != -1) {
            count++;
            checkLimit();
        }
        return res;
    }

    /**
     * Reads up to <code>len</code> bytes of data from this input stream
     * into an array of bytes. If <code>len</code> is not zero, the method
     * blocks until some input is available; otherwise, no
     * bytes are read and <code>0</code> is returned.
     * <p>
     * This method simply performs <code>in.read(b, off, len)</code>
     * and returns the result.
     *
     * @param      b     the buffer into which the data is read.
     * @param      off   The start offset in the destination array
     *                   <code>b</code>.
     * @param      len   the maximum number of bytes read.
     * @return     the total number of bytes read into the buffer, or
     *             <code>-1</code> if there is no more data because the end of
     *             the stream has been reached.
     * @throws  NullPointerException If <code>b</code> is <code>null</code>.
     * @throws  IndexOutOfBoundsException If <code>off</code> is negative,
     * <code>len</code> is negative, or <code>len</code> is greater than
     * <code>b.length - off</code>
     * @throws  IOException  if an I/O error occurs.
     * @see        java.io.FilterInputStream#in
     */
    @Override
    public int read(byte[] b, int off, int len) throws IOException {
        int res = super.read(b, off, len);
        if (res > 0) {
            count += res;
            checkLimit();
        }
        return res;
    }

    /**
     * Returns, whether this stream is already closed.
     *
     * @return True, if the stream is closed, otherwise false.
     * @throws IOException An I/O error occurred.
     */
    @Override
    public boolean isClosed() throws IOException {
        return closed;
    }

    /**
     * Closes this input stream and releases any system resources
     * associated with the stream.
     * This
     * method simply performs <code>in.close()</code>.
     *
     * @throws  IOException  if an I/O error occurs.
     * @see        java.io.FilterInputStream#in
     */
    @Override
    public void close() throws IOException {
        closed = true;
        super.close();
    }
}

You should subclass this and override the raiseError method.

Mateusz Stefek
  • 173
  • 1
  • 6
1

Much more convenient and faster solution to check size of input stream  

    FileChannel chanel = (FileChannel) Channels.newChannel(inputStream);
    MappedByteBuffer buffer = chanel.map(FileChannel.MapMode.READ_ONLY, 0, chanel.size());

    System.out.println(buffer.capacity()); // bytes
aolo23
  • 19
  • 1
  • 2
0

I like the ByteArrayOutputStream-based solution, I cant see why it cant work

public void copy(InputStream input, Path target) throws IOException {
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    BufferedInputStream bis = new BufferedInputStream(input);
    for (int b = 0; (b = bis.read()) != -1;) {
        if (bos.size() > BUFFER_SIZE) {
            throw new IOException();
        }
        bos.write(b);
    }
    Files.write(target, bos.toByteArray());
}
Evgeniy Dorofeev
  • 133,369
  • 30
  • 199
  • 275
  • Sure, it *can* work. I'd just question the *utility* of buffering the file in memory like that. Unless we assume that there are going to be a significant percentage of the "too big" case, it is cheaper and simpler to just write straight to the output file ... and delete it in the error case as the OP is doing. – Stephen C Mar 16 '13 at 06:23
  • Vote up. My concern is that the limit will be much higher in the future, as result ByteArrayOutputStream will consume hugh memory. – 卢声远 Shengyuan Lu Mar 16 '13 at 06:28
  • Why waste all that memory? – user207421 Jul 21 '16 at 01:58