It seems that it is not possible to do POST with body >10K with scala play ws framework

Question

It seems that it is not possible to do POST with body >10K

If I do:

WS.url(url).post("content more than 10K")

I've got clipped body. Exactly 10K. How can I avoid this limitation?

Answer 1

The usual approach is to POST (or PUT) large documents via a stream. This used to be much harder than it should be but if you're willing to use Play 2.3 it has improved. Check out the section "Play WS" at the What's new? page. The underlying AsyncHttpClient is probably what you need -- Play used not to let you get at this and even now you need to use another library (Dispatch) after pulling the object out of WS.

Basically, as the above doc shows, you instantiate a NingWSClient, use .underlying to pull out the AsyncHttpClient, then make async calls on that to post your stream. The second of these two historical threads gives you some more hints but figuring out how to make the relevant calls on AsyncHttpClient will itself be interesting. I'll see if I can figure it out tonight but I thought this might be useful to you now.

Play WS (2.2.1): post/put large request -- the accepted answer shows you how to get at the underlying HTTP client. The second answer foreshadows the Play 2.3 feature above.

Simple and concise HTTP client library for Scala -- the accepted answer talks about Dispatch, the Scala wrapper for Ning's async HTTP client, which is what the first thread's accepted answer refers to.

The Dispatch doc I must say deserves its reputation. Unless somebody comes up with a simple fix for your problem, or if you need to POST large documents a lot, I really recommend the 2.3 route and Dispatch.

EDIT: Here's an example of how to do this without using the WS APIs at all, but using the Ning client directly. I used the Ning packages that come with the play-ws library from Play 2.4.0-M1 because it was what I had handy. You may be able to get the Ning libraries separately and stick to whatever Play version you are using. This is adapted from a streaming GET example I found

import scala.concurrent.Promise
import com.ning.http.client._
import com.ning.http.client.Response
import com.ning.http.client.AsyncCompletionHandler
import scala.concurrent.ExecutionContext.Implicits.global
import java.io.ByteArrayInputStream

object AsyncWS {
  def main(args: Array[String]): Unit = {
    val config = new AsyncHttpClientConfig.Builder()
    val client = new AsyncHttpClient(config.build())

    // this will send back your request including the body
    val url = "http://httpbin.org/post"

    // testing the basic approach by still reading from a string -- this
    // should work if you can fir your text into a string, otherwise
    // you'll need to write some stream magic
    val testInputStream = 
      new ByteArrayInputStream("s short test string".getBytes)

    val requestBuilder = client.preparePost(url)
    requestBuilder.setBody(testInputStream)

    val request = requestBuilder.build()
    var result = Promise[Int]()
    client.executeRequest(request, new AsyncCompletionHandler[Response]() {
      override def onCompleted(response: Response) = {
        println("got an answer")
        val is  = response.getResponseBodyAsStream()
        val in = scala.io.Source.fromInputStream(is)
        in.getLines().foreach(println(_))
        in.close()
        result.success(response.getStatusCode())
        response
      }

      override def onThrowable(t: Throwable) {
        result.failure(t)
      }
    })
    val getFuture = result.future

    getFuture.map { status =>
      println("result status = " + status)
    }.recover {
      case e: Throwable => {
        println("error e = " + e)
      }
    }
  }
}