Scala http operations

Question

How do I perform the following in Scala?

• HTTP Get
• HTTP Get With custom headers
• HTTP Post

Answer 1

You could try out Dispatch. A little difficult to grasp at first, but after a while I've started to like it. It works on top of HttpClient.

import dispatch.Http
import Http._
// Get
Http(url("http://youruri.com/yo") >>> System.out)
// Get with header
Http(url("http://youruri.com/yo") <:< Map("Accept" -> "application/json") >>> System.out)
// Post
Http(url("http://youruri.com/yo") << yourPostData >|)

Answer 2

You can simply use java.net.URL to send HTTP GET and HTTP POST requests. You can also set HTTP request headers on the HttpURLConnection like this:

val con = url.openConnection.asInstanceOf[HttpURLConnection]
con.setRequestProperty("Header", "Value")

I have written myself a utility class which does exactly this. You can see it here:

https://github.com/gruenewa/gruenewa-misc/blob/master/gruenewa-wsclient/src/main/scala/gruenewa/wsclient/Service.scala

Answer 3

this is my own implementation of a simple Http client including cookies management. Maybe it will be useful for you. But I'm not sure if header modification is directly possible (it may require your own implementation of URLConnection).

import java.io.OutputStreamWriter
import java.net.{URLConnection, URL}

class Http(userAgent: String,
           encoding: String,
           HttpRequestTimeout: Int = 15000) {

  import collection.JavaConversions._
  import Implicits.wrapInputStream
  import java.net.URLEncoder.encode

  var cookies = Map[String, String]()

  private def loadCookies(conn: URLConnection) {
    for ((name, value) <- cookies) conn.setRequestProperty("Cookie", name + "=" + value)
  }

  private def saveCookies(conn: URLConnection) {
    conn.getHeaderFields.lift("Set-Cookie") match {
      case Some(cList) => cList foreach { c =>
        val (name,value) = c span { _ != '=' }
        cookies += name -> (value drop 1)
      }
      case None =>
    }
  }

  private def encodePostData(data: Map[String, String]) =
    (for ((name, value) <- data) yield encode(name, encoding) + "=" + encode(value, encoding)).mkString("&")

  def Get(url: String) = {
    val u = new URL(url)
    val conn = u.openConnection()
    conn.setRequestProperty("User-Agent", userAgent)
    conn.setConnectTimeout(HttpRequestTimeout)

    loadCookies(conn)

    conn.connect

    saveCookies(conn)

    conn.getInputStream.mkString
  }

  def Post(url: String, data: Map[String, String]) = {
    val u = new URL(url)
    val conn = u.openConnection

    conn.setRequestProperty("User-Agent", userAgent)
    conn.setConnectTimeout(HttpRequestTimeout)

    loadCookies(conn)

    conn.setDoOutput(true)
    conn.connect

   val wr = new OutputStreamWriter(conn.getOutputStream())
    wr.write(encodePostData(data))
    wr.flush
    wr.close


    saveCookies(conn)

    conn.getInputStream.mkString
  }
}

Answer 4

While I appreciate the Dispatch library for all it's worth, the syntax still confuses me a bit.

Someone directed me to scalaj-http the other day which seems a little easier

Answer 5

Regarding simply GETting data from URL. If you don't want to use external sources, then:

  val in = scala.io.Source.fromURL("http://some.your.url/params?start&here", 
                                   "utf-8")
  for (line <- in.getLines)
    println(line)  

For all other stuff, you can choose any method you like from answers above.

Answer 6

Based on @Antonin Brettsnajdr's answer, a simply version of uploading a file using POST

val conn = new URL("http://myserver.appspot.com/upload").openConnection()
conn.setDoOutput(true)
conn.connect
val input = new FileInputStream(file)
val buffer = new Array[Byte](2 * 1024 * 1024)
Stream.continually(input.read(buffer)).takeWhile(_ != 1).foreach(conn.getOutputStream.write(_))

Answer 7

You could use spray-client. The documentation is lacking (it took me some digging to find out how to make GET requests with query parameters) but it's a great option if you are already using spray. We're using it at AI2 over dispatch because the operators are less symbolic and we're already using spray/actors.

import spray.client.pipelining._

val url = "http://youruri.com/yo"
val pipeline: HttpRequest => Future[HttpResponse] = addHeader("X-My-Special-Header", "fancy-value") ~ sendReceive

// Get with header
pipeline(Get(url)) map (_.entity.asString) onSuccess { case msg => println(msg) }

// Get with header and parameters
pipeline(Get(Uri(url) withParams ("param" -> paramValue)) map (_.entity.asString) onSuccess { case msg => println(msg) }

// Post with header
pipeline(Post(url, yourPostData)) map (_.entity.asString) onSuccess { case msg => println(msg) }

Answer 8

I've used Dispatch, Spray Client and the Play WS Client Library...None of them were simply to use or configure. So I created a simpler HTTP Client library which lets you perform all the classic HTTP requests in simple one-liners.

See an example:

import cirrus.clients.BasicHTTP.GET

import scala.concurrent.Await
import scala.concurrent.duration._

object MinimalExample extends App {

  val html = Await.result(Cirrus(GET("https://www.google.co.uk")), 3 seconds)

  println(html)
}

... produces ...

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-GB">...</html>

The library is called Cirrus and is available via Maven Central

libraryDependencies += "com.github.godis" % "cirrus_2.11" % "1.4.1"

The documentation is available on GitHub

https://github.com/Godis/Cirrus