How can I process a file uploaded through an HTML form in Spray/Scala/Java?

Question

I have an application written using Spray, and I have a page which has an <input type="file" name="foo"> form element that gets POSTed to /fileUpload.

I have a Spray route set up to listen to the path /fileUpload using this code:

path("fileUpload") {
  get { ctx => {
    val request: HttpRequest = ctx.request
    //Process the file, somehow with request?
    ctx.complete("File Uploaded")
  }}
}

I can't figure out how to get the POST body and get a handle on the file, and I can't find any examples online.

It must be possible to receive a file and process it with Spray or even through simple Scala or Java, but I don't know how to do it.

Can anybody help?
Thanks!

I, too, was puzzled why there isn't a mention for this in the manuals, no samples, and so on.. so there's at least two of us. Would really expect Spray to have some helpers for this. — akauppi, Jul 30 '14 at 22:37

iwein · Accepted Answer · 2014-08-04T07:11:52.500

9

It's possible with Spray, although I haven't checked if streaming works properly. I fiddled a bit and got this working:

  post {
    content(as[MultipartFormData]) {
      def extractOriginalText(formData: MultipartFormData): String = {
        formData.parts.mapValues { (bodyPart) =>
          bodyPart.content.map{
            (content) => new String(content.buffer)
          }
        }.values.flatten.foldLeft("")(_ + _)
      }
      formData =>
        _.complete(
          extractOriginalText(formData)
        );
    }

If you upload a plain text file to a service that has this code in it, it coughs the original text back up. I've got it running together with an ajax upload; it should also work with an old fashioned file upload form.

It seems to me that there must be an easier way to do this, particularly the deep nesting of the content is rather clunky. Let me know if you find a simplification.

UPDATE (thx akauppi):

entity(as[MultipartFormData]) { formData => 
  complete( formData.fields.map { _.entity.asString }.flatten.foldLeft("")(_ + _) ) 
}

edited Aug 04 '14 at 07:11

answered Sep 16 '12 at 19:25

iwein

25,788
10
70
111

Also, I found this mailing list mention https://groups.google.com/forum/#!topic/spray-user/_ZcXPqYM7Cs "so far there is no dedicated uploading support, but it should be quite easy to supply the few remaining pieces yourself." from Mathias, the author of Spray (from 2012) – akauppi Jul 30 '14 at 23:04
akauppi feel free to suggest an edit, I like up to date answers. Did you compile it? – iwein Jul 31 '14 at 12:24
Your solution does not work for binary files, since it does the concatenation as strings. I'll send an answer that works also with binaries. – akauppi Aug 01 '14 at 08:56
SO seems to be eating the underscores. Anything we can do about that? – akauppi Aug 02 '14 at 08:47
@akauppi use backticks: `_let's see_` – iwein Aug 04 '14 at 07:12
thx, will do in the future. deleted the original comment since without the ticks it was so weird. – akauppi Aug 04 '14 at 10:37

score 4 · Answer 2 · edited May 23 '17 at 12:16

I ended up using the code below. Wasn't too hard, but there really should have been a Spray sample available somewhere about this.

multipart/form-data forms must always be used (instead of the traditional application/x-www-form-urlencoded) if binary uploads are involved. More details here.

My requirements were:

need to upload binary files of reasonably large size
wanted to have metadata as fields (not embedded in URL or the upload's filename)

Some questions:

is the way I'm managing errors the "best" way?

It's in the essence of REST API design to treat the client as a "human" (in debugging, we are), giving meaningful error messages in case something is wrong with the message.

post {
  // Note: We cannot use a regular 'return' to provide a routing mid-way. The last item matters, but we can
  // have a 'var' which collects the correct success / error info into it. It's a write-once variable.
  //
  var ret: Option[Route] = None

  // Multipart form
  //
  // To exercise this:
  //  $ curl -i -F "file=@filename.bin" -F "computer=MYPC" http://localhost:8080/your/route; echo
  //
  entity(as[MultipartFormData]) { formData =>
    val file = formData.get("file")
      // e.g. Some(
      //        BodyPart( HttpEntity( application/octet-stream, ...binary data...,
      //                    List(Content-Type: application/octet-stream, Content-Disposition: form-data; name=file; filename=<string>)))
    log.debug( s".file: $file")

    val computer = formData.get("computer")
    // e.g. Some( BodyPart( HttpEntity(text/plain; charset=UTF-8,MYPC), List(Content-Disposition: form-data; name=computer)))
    log.debug( s"computer: $computer" )

    // Note: The types are mentioned below simply to make code comprehension easier. Scala could deduce them.
    //
    for( file_bodypart: BodyPart <- file;
        computer_bodypart: BodyPart <- computer ) {
      // BodyPart: http://spray.io/documentation/1.1-SNAPSHOT/api/index.html#spray.http.BodyPart

      val file_entity: HttpEntity = file_bodypart.entity
        //
        // HttpEntity: http://spray.io/documentation/1.1-SNAPSHOT/api/index.html#spray.http.HttpEntity
        //
        // HttpData: http://spray.io/documentation/1.1-SNAPSHOT/api/index.html#spray.http.HttpData

      log.debug( s"File entity length: ${file_entity.data.length}" )

      val file_bin= file_entity.data.toByteArray
      log.debug( s"File bin length: ${file_bin.length}" )

      val computer_name = computer_bodypart.entity.asString    //note: could give encoding as parameter
      log.debug( s"Computer name: $computer_name" )

      // We have the data here, pass it on to an actor and return OK
      //
      ...left out, application specific...

      ret = Some(complete("Got the file, thanks.")) // the string doesn't actually matter, we're just being polite
    }

    ret.getOrElse(
      complete( BadRequest, "Missing fields, expecting file=<binary>, computer=<string>" )
    )
  }
}

That code looks like my thumb after the accident in 93, but you've got it working, and that's worth an upvote. — iwein, Aug 01 '14 at 20:16
The process did feel like hammering my own thumb. :P Feedback on how I could make the code better is appreciated. — akauppi, Aug 02 '14 at 08:44

score 2 · Answer 3 · answered Sep 22 '11 at 19:23

Ok, after trying to write a Spray Unmarshaller for multipart form data, I decided to just write a scala HttpServlet that would receive the form submission, and used Apache's FileUpload library to process the request:

class FileUploadServlet extends HttpServlet {

  override def doPost(request: HttpServletRequest, response: HttpServletResponse) {
    val contentType = request.getContentType
    val boundary = contentType.substring(contentType.indexOf("boundary=")+9)
    val multipartStream = new MultipartStream(request.getInputStream, boundary)

    // Do work with the multipart stream
  }

}

I found using commons-fileupload excruciating. Combining it with Spray is insane imho. — iwein, Sep 16 '12 at 19:29

score 1 · Answer 4 · answered Oct 13 '15 at 14:34

To grab the posted (possibly binary) file, and stick it somewhere temporarily, I used this:

  post {
    entity(as[MultipartFormData]) {
      formData => {
        val ftmp = File.createTempFile("upload", ".tmp", new File("/tmp"))
        val output = new FileOutputStream(ftmp)
        formData.fields.foreach(f => output.write(f.entity.data.toByteArray ) )
        output.close()
        complete("done, file in: " + ftmp.getName())
      }
    }
  }

How can I process a file uploaded through an HTML form in Spray/Scala/Java?

4 Answers4

Linked