How can I convert any nested combination of maps, seqs, and basic types to JSON in Scala?

Question

In Python I can arbitrarily nest lists and dictionaries and use a mixture of types and then simply call json.dumps(x) and get the result I need.

I can't find anything like this in Scala. All the libraries I come across seem to insist on static typing and compile-time checks. I would rather give that up for simplicity. It seems that it should be possible to dynamically check the types of the inputs.

As a simple example, I would like to be able to do something like:

toJson(Map("count" -> 1,
           "objects" -> Seq(Map("bool_val" -> true,
                                "string_val" -> "hello"))))

which would output a string containing:

{"count": 1, "objects": [{"bool_val": true, "string_val": "hello"}]}

EDIT: Here is what happens when I try a couple of libraries:

scala> import upickle.default._
import upickle.default._

scala> write(Seq(1, 2))
res0: String = [1,2]

scala> write(Seq(1, "2"))
<console>:15: error: Couldn't derive type Seq[Any]
       write(Seq(1, "2"))
            ^

scala> import spray.json._
import spray.json._

scala> import spray.json.DefaultJsonProtocol._
import spray.json.DefaultJsonProtocol._

scala> Seq(1, 2).toJson
res2: spray.json.JsValue = [1,2]

scala> Seq(1, "2").toJson
<console>:21: error: Cannot find JsonWriter or JsonFormat type class for Seq[Any]
       Seq(1, "2").toJson
                   ^

I tried creating my own general protocol for spray but I'm getting weird results:

import spray.json._

object MyJsonProtocol extends DefaultJsonProtocol {

  implicit object AnyJsonWriter extends JsonFormat[Any] {
    def write(x: Any) = x match {
      case n: Int => JsNumber(n)
      case n: Long => JsNumber(n)
      case d: Double => JsNumber(d)
      case s: String => JsString(s)
      case b: Boolean if b => JsTrue
      case b: Boolean if !b => JsFalse
      case m: Map[Any, Any] => m.toJson
      case p: Product => p.productIterator.toList.toJson  // for tuples
      case s: Seq[Any] => s.toJson
      case a: Array[Any] => a.toJson
    }

    override def read(json: JsValue) = ???
  }

}

import MyJsonProtocol._

val objects = Seq(Map(
  "bool_val" -> true,
  "string_val" -> "hello"))

objects.toJson.toString
// [{"bool_val":true,"string_val":"hello"}]

Map(
  "count" -> 1,
  "objects" -> objects).toJson.toString
// {"count":1,"objects":[{"bool_val":true,"string_val":"hello"},[]]}
//                          I don't know where this comes from: ^

Maybe this answer will provide some info https://stackoverflow.com/questions/8054018/what-json-library-to-use-in-scala. But there are too many libraries. — aristotll, Aug 14 '17 at 15:57
Scala is all about strict typing and compile-time safety. If you want to give that up "for simplicity", why not just write in python to begin with? — Dima, Aug 14 '17 at 16:58
But, yeah, you can do what you are describing in scala, it's possible, just not very useful. I am not sure what you are asking exactly. — Dima, Aug 14 '17 at 17:04
@Dima Among other things, I have a `List[Map[String, Any]]` from a spark dataframe whose schema is determined by user input, so I can't do much proper typing. — Alex Hall, Aug 14 '17 at 20:44
So, what's the question? Whatever library you use to generate json should be able to deal with that type. — Dima, Aug 15 '17 at 00:57
If the schema is determined by user input, then you should ask for some sort of `JsonEncoder`/`JsonDecoder` for the user-defined types. That is how most libraries that allow this kind of stuff work. — HTNW, Aug 15 '17 at 01:18
@Dima no, `Any` freaks libraries out, that's why I'm here. `Seq(1, "2")` is enough of a problem. — Alex Hall, Aug 15 '17 at 07:43
@HTNW no, that's not the kind of user I mean. Point is, this isn't an XY problem, please let that go. — Alex Hall, Aug 15 '17 at 07:44
@AlexHall You gotta be (_a lot_) more specific with that: what libraries are you using, what do you mean by "freaks out" etc.? Have you read about mcve? http://stackoverflow.com/help/mcve — Dima, Aug 15 '17 at 12:29
I don't know spray ... what you are describing looks like a bug. I recommend jackson with scala module. Perhaps, you'll have better luck with that. Judging from your sample, it would also require A LOT less boilerplate to set up. — Dima, Aug 15 '17 at 14:16
@Dima thanks for the pointer, I managed to do it with Jackson (see answer). — Alex Hall, Aug 15 '17 at 14:33

HTNW · Answer 1 · 2017-08-14T19:19:51.973

If you just want to throw away the type system and cast everything:

import math._ // For BigDecimal and the associated implicit conversions

// Dispatch to another overload if possible, or crash and burn with MatchError
def toJson(any: Any): String = any match {
  case null            => "null"
  case obj: Map[_, _]  => toJson(obj.asInstanceOf[Map[String, Any]])
  case arr: Seq[_]     => toJson(arr)
  case str: String     => toJson(str)
  case  bl: Boolean    => toJson(bl)
  case  bd: BigDecimal => toJson(bd)
  // v Convert to BD so toJson(Number) doesn't need to be duplicated
  case   l: Long       => toJson(l: BigDecimal)
  case   i: Int        => toJson(i: BigDecimal)
  case   s: Short      => toJson(s: BigDecimal)
  case   c: Char       => toJson(c: BigDecimal)
  case   b: Byte       => toJson(b: BigDecimal)
  case   f: Float      => toJson(f: BigDecimal)
  case   d: Double     => toJson(d: BigDecimal)
}

def toJson(obj: Map[String, Any]): String =
  // Build a list of "key": "value" entries
  obj.foldRight(Seq.empty[String]) { case ((prop, value), building) =>
    s"${toJson(prop)}: ${toJson(value)}" +: building
  }.mkString("{ ", ", ", " }")
  // Wrap in { ... } and add ", "s

def toJson(arr: Seq[Any]): String =
  // Build list of JSON strings
  arr.foldRight(Seq.empty[String]) { (value, building) =>
    toJson(value) +: building
  }.mkString("[ ", ", ", " ]")
  // Wrap in [ ... ] and add ", "s

// Process each character, one by one
def toJson(str: String): String = str.flatMap {
  // Common escapes
  case '\\' => "\\\\"
  case '"'  => "\\\""
  case '\b' => "\\b"
  case '\f' => "\\f"
  case '\n' => "\\n"
  case '\r' => "\\r"
  case '\t' => "\\t"
  // All the other control characters
  case ctrl if ctrl < ' ' => f"\\u${ctrl}%04x"
  // Nothing special: leave unchanged
  case c  => c.toString
}.mkString("\"", "", "\"")
// Wrap in "..."

def toJson(bl: Boolean): String = bl.toString

def toJson(bd: BigDecimal): String = bd.toString

Please note that this is a very un-Scala thing to do. Python doesn't have a type system like Scala, and that's why you are used to runtime-checked code like this. But Scala isn't like that, and you should be working with the typer, not against it. For one, creating maps and seqs containing Any like this is a massive red flag. Consider using case classes to hold data. To that purpose, a lot of the JSON libraries for Scala support auto-deriving JSON en-/de-coders, usually with shapeless.Generic.

This is impressive, I wasn't expecting a solution from scratch, but I don't feel great using it. I'd prefer to use a well established and tested library so that I know that all corner cases in the spec are handled. It also doesn't look very fast (e.g. `str.flatMap`). — Alex Hall, Aug 15 '17 at 13:58

score 0 · Accepted Answer · answered Aug 15 '17 at 14:32

Based on https://coderwall.com/p/o--apg/easy-json-un-marshalling-in-scala-with-jackson:

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper

val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)

def toJson(value: Any): String = {
  mapper.writeValueAsString(value)
}

How can I convert any nested combination of maps, seqs, and basic types to JSON in Scala?

2 Answers2