Multipart File Upload on Google Appengine using jersey-1.7

Question

I wrote an application on Google Appengine with Jersey to handle simple file uploading. This works fine when it was on jersey 1.2. In the later versions (current 1.7) @FormDataParam is introduced to handle multipart/form inputs. I am using jersey-multipart and the mimepull dependency. It seems that the new way of doing it is creating temporary files in appengine which we all know is illegal...

Am I missing something or doing something wrong here since Jersey is now supposedly compatible with AppEngine?

@POST 
@Path("upload") 
@Consumes(MediaType.MULTIPART_FORM_DATA) 
public void upload(@FormDataParam("file") InputStream in) { .... }

The above will fail when called with these exceptions...

/upload
java.lang.SecurityException: Unable to create temporary file
    at java.io.File.checkAndCreate(File.java:1778)
    at java.io.File.createTempFile(File.java:1870)
    at java.io.File.createTempFile(File.java:1907)
    at org.jvnet.mimepull.MemoryData.createNext(MemoryData.java:87)
    at org.jvnet.mimepull.Chunk.createNext(Chunk.java:59)
    at org.jvnet.mimepull.DataHead.addBody(DataHead.java:82)
    at org.jvnet.mimepull.MIMEPart.addBody(MIMEPart.java:192)
    at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:235)
    at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:176)
    at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:101)
    at com.sun.jersey.multipart.impl.MultiPartReaderClientSide.readMultiPart(MultiPartReaderClientSide.java:177)
    at com.sun.jersey.multipart.impl.MultiPartReaderServerSide.readMultiPart(MultiPartReaderServerSide.java:80)
    at com.sun.jersey.multipart.impl.MultiPartReaderClientSide.readFrom(MultiPartReaderClientSide.java:139)
    at com.sun.jersey.multipart.impl.MultiPartReaderClientSide.readFrom(MultiPartReaderClientSide.java:77)
    at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
    at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:538)

Anyone have a clue? Is there a way to do thing while preventing mimepull from creating the temporary file?

score 17 · Answer 1 · answered Jun 10 '11 at 09:20

17

For files beyond its default size, multipart will create a temporary file. To avoid this — creating a file is impossible on gae — you can create a jersey-multipart-config.properties file in the project's resources folder and add this line to it:

bufferThreshold = -1

Then, the code is the one you gave:

@POST
@Consumes(MediaType.MULTIPART_FORM_DATA)
public Response post(@FormDataParam("file") InputStream stream, @FormDataParam("file") FormDataContentDisposition disposition) throws IOException {
  post(file, disposition.getFileName());
  return Response.ok().build();
}

answered Jun 10 '11 at 09:20

yves amsellem

7,106
5
44
68

1

Any idea how to do this programmatically? – gk5885 Sep 28 '12 at 00:16
@gk5885 By "programmatically" you mean without Jersey/JAX-RS? – yves amsellem Sep 28 '12 at 13:43
I mean how to set the bufferThrehold in code rather than via a properties file. – gk5885 Oct 06 '12 at 17:01
7

Thanks for the solution. Just an FYI, if you're using Jersey 2+ you need to use `jersey.config.multipart.bufferThreshold` as your property name. – David Welch Sep 03 '13 at 19:20
4

Can you please help me with where the `resources` folder should be. Is it in `\war` or `\WEB-INF` or somewhere else? Thanks. – Floris Sep 11 '14 at 14:54
Thanks a lot , this has made my day ... I was struggling with this issue for quite some time. Thanks a lot again – Indraneel Aug 02 '16 at 03:40
This seems like the answer – I.Tyger Oct 07 '16 at 15:48
Is there any way to set a file size limit ? – Adrian Bob Jun 05 '18 at 22:12

score 11 · Answer 2 · answered Sep 15 '14 at 12:51

11

For the benefit of those struggling when using Eclipse with GPE (Google Plugin for Eclipse) I give this slightly modified solution derived from @yves' answer.

I have tested it with App Engine SDK 1.9.10 and Jersey 2.12. It will not work with App Engine SDK 1.9.6 -> 1.9.9 amongst others due to a different issue.

Under your \war\WEB-INF\classes folder create a new file called jersey-multipart-config.properties. Edit the file so it contains the line jersey.config.multipart.bufferThreshold = -1.

Note that the \classes folder is hidden in Eclipse so look for the folder in your operating system's file explorer (e.g. Windows Explorer).

Now, both when the multipart feature gets initialized (on Jersey servlet initialization) and when a file upload is done (on Jersey servlet post request) the temp file will not be created anymore and GAE won't complain.

answered Sep 15 '14 at 12:51

Floris

1,082
10
26

1

Thanks man! `jersey.config.multipart.` before the `bufferThreshold` fixed it! ;) – DominikAngerer Mar 23 '15 at 12:35
1

For anyone wanting to do this in eclipse: Yes, the `classes` folder doesn't show up in the Package Explorer view, but you can still access it by using the _Navigator_ view instead of _Package Explorer_. (From there, you could even right-click and do "Show in Remote Systems view" if you wanted to explore the underlying file system.) – Amos M. Carpenter Dec 04 '15 at 01:26
You got Jersey 2 working with App Engine? I thought it depended on servlet-api:1.3.x. I know this is super old, but I'd be interested to see your dependency tree... – ndtreviv Oct 03 '16 at 11:28

kavai77 · Answer 3 · 2015-07-10T03:46:50.357

It is very important to put the file jersey-multipart-config.properties under WEB-INF/classes inside the WAR.

Usually in a WAR file structure you put the config files (web.xml, appengine-web.xml) into WEB-INF/, but here you need to put into WEB-INF/classes.

Example Maven configuration:

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-war-plugin</artifactId>
            <version>2.4</version>
            <configuration>
                <archiveClasses>true</archiveClasses>
                <webResources>
                    <resource>
                        <directory>${basedir}/src/main/webapp/WEB-INF</directory>
                        <filtering>true</filtering>
                        <targetPath>WEB-INF</targetPath>
                    </resource>
                    <resource>
                        <directory>${basedir}/src/main/resources</directory>
                        <targetPath>WEB-INF/classes</targetPath>
                    </resource>
                </webResources>
            </configuration>
        </plugin>

And your project structure can look like:

Project Structure

Content of jersey-multipart-config.properties with Jersey 2.x:

jersey.config.multipart.bufferThreshold = -1

score 1 · Answer 4 · edited Sep 24 '14 at 08:35

i've found solution to programmatically avoid to use temporary file creation (very useful for GAE implementation)

My solution consist of creating a new MultiPartReader Provider ... below my code

  @Provider
    @Consumes("multipart/*")
    public class GaeMultiPartReader implements MessageBodyReader<MultiPart> {

    final Log logger = org.apache.commons.logging.LogFactory.getLog(getClass());

    private final Providers providers;

    private final CloseableService closeableService;

    private final MIMEConfig mimeConfig;

    private String getFixedHeaderValue(Header h) {
        String result = h.getValue();

        if (h.getName().equals("Content-Disposition") && (result.indexOf("filename=") != -1)) {
            try {
                result = new String(result.getBytes(), "utf8");
            } catch (UnsupportedEncodingException e) {            
                final String msg = "Can't convert header \"Content-Disposition\" to UTF8 format.";
                logger.error(msg,e);
                throw new RuntimeException(msg);
            }
        }

        return result;
    }

    public GaeMultiPartReader(@Context Providers providers, @Context MultiPartConfig config,
        @Context CloseableService closeableService) {
        this.providers = providers;

        if (config == null) {
            final String msg = "The MultiPartConfig instance we expected is not present. "
                + "Have you registered the MultiPartConfigProvider class?";
            logger.error( msg );
            throw new IllegalArgumentException(msg);
        }
        this.closeableService = closeableService;

        mimeConfig = new MIMEConfig();
        //mimeConfig.setMemoryThreshold(config.getBufferThreshold());
        mimeConfig.setMemoryThreshold(-1L); // GAE FIX
    }

    @Override
    public boolean isReadable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        return MultiPart.class.isAssignableFrom(type);
    }

    @Override
    public MultiPart readFrom(Class<MultiPart> type, Type genericType, Annotation[] annotations, MediaType mediaType,
        MultivaluedMap<String, String> headers, InputStream stream) throws IOException, WebApplicationException {
        try {
            MIMEMessage mm = new MIMEMessage(stream, mediaType.getParameters().get("boundary"), mimeConfig);

            boolean formData = false;
            MultiPart multiPart = null;

            if (MediaTypes.typeEquals(mediaType, MediaType.MULTIPART_FORM_DATA_TYPE)) {
                multiPart = new FormDataMultiPart();
                formData = true;
            } else {
                multiPart = new MultiPart();
            }

            multiPart.setProviders(providers);

            if (!formData) {
                multiPart.setMediaType(mediaType);
            }

            for (MIMEPart mp : mm.getAttachments()) {
                BodyPart bodyPart = null;

                if (formData) {
                    bodyPart = new FormDataBodyPart();
                } else {
                    bodyPart = new BodyPart();
                }

                bodyPart.setProviders(providers);

                for (Header h : mp.getAllHeaders()) {
                    bodyPart.getHeaders().add(h.getName(), getFixedHeaderValue(h));
                }

                try {
                    String contentType = bodyPart.getHeaders().getFirst("Content-Type");

                    if (contentType != null) {
                        bodyPart.setMediaType(MediaType.valueOf(contentType));
                    }

                    bodyPart.getContentDisposition();
                } catch (IllegalArgumentException ex) {
                    logger.error( "readFrom error", ex );
                    throw new WebApplicationException(ex, 400);
                }

                bodyPart.setEntity(new BodyPartEntity(mp));
                multiPart.getBodyParts().add(bodyPart);
            }

            if (closeableService != null) {
                closeableService.add(multiPart);
            }

            return multiPart;
        } catch (MIMEParsingException ex) {
            logger.error( "readFrom error", ex );
            throw new WebApplicationException(ex, 400);
        }
    }

}

score 0 · Answer 5 · answered Aug 02 '13 at 05:04

We experienced a similar problem, Jetty wouldn't let us upload files more than 9194 bytes, (all of a sudden - one day), we realised afterwards that someone had taken our user access from /tmp, which corresponds to java.io.tmpdir on some linux versions, so Jetty couldn't store the uploaded file there, and we got a 400 error.

Multipart File Upload on Google Appengine using jersey-1.7

5 Answers5

Linked