How can I deserialize an non-fixed array of jsons using Circe's manual decoder?

Question

I have a JSON that looks like:

{
  "data": [
    {
      "id": "1",
      "email": "hello@world.com",
      "name": "Mr foo",
      "roles": [
        "Chief Bar Officer"
      ],
      "avatar_url": null,
      "phone_number": null
    },
    {
      "id": "2",
      "email": "bye@world.com",
      "name": "Mr baz",
      "roles": [
        "Chief Baz Officer"
      ],
      "avatar_url": null,
      "phone_number": null
    }
  ]
}

I am mainly interested in parsing/deserializing the data list, and I would like to do that manually (I prefer the manual way for some mysterious reason).

In case this is relevant, I am using sttp's circe library sttp.client.circe._ with the intention of parsing incoming data from get requests directly into Json using asJson.

A get sttp request looks something like:

val r1 = basicRequest
    .get(uri"https://woooo.woo.wo/v1/users")
    .header("accept", "application/json")
    .header("Authorization", "topsecret"
    .response(asJson[SomeClass])

This is what I have tried so far:

// Define the case class
case class User(
    id: String,
    email: String,
    name: String,
    roles: List[String],
    avatar_url: Option[String],
    phone_number: Option[String]
)

// Define the manual deserializer

case object User {

  implicit val userDecoder: Decoder[User] = (hCursor: HCursor) => {
    val data = hCursor.downField("data").downArray
    for {
      id <- data.get[String]("id")
      email <- data.get[String]("email")
      name <- data.get[String]("name")
      roles <- data.get[List[String]]("roles")
      avatarUrl <- data.get[Option[String]]("avatarUrl")
      phoneNumber <- data.get[Option[String]]("phoneNumber")
    } yield User(id, email, name, roles, avatarUrl, phoneNumber)
  }
}

The problem with my approach (I think) is that .downArray makes me only serialize the first User in the array of Users.

My objective is to be able to have some sequence of users (something like List[User] maybe), but at the moment I only end up deserializing one user in the array.

It is worth mentioning that the "data" array, does not contain a fixed-number of users, and every api call could result in a different number of users.

score 3 · Accepted Answer · answered Aug 06 '20 at 12:15

Thanks to the help of Travis Brown and the circe Gitter community for helping me figure this one out.

I'm quoting Travis here:

it would be better to build up the instance you need to parse the top-level JSON object compositionally… i.e. have a Decoder[User] that only decodes a single user JSON object, and then use Decoder[List[User]].at("data") or something similar to decode the top-level JSON object containing the data field with the JSON array.

I have ended up with an implementation that looks something like:

case class Users(users: List[User])

case object User {

  implicit val usrDecoder: Decoder[User] = (hCursor: HCursor) => {

    for {
      id <- hCursor.get[String]("id")
      email <- hCursor.get[String]("email")
      name <- hCursor.get[String]("name")
      roles <- hCursor.get[List[String]]("roles")
      avatarUrl <- hCursor.get[Option[String]]("avatarUrl")
      phoneNumber <- hCursor.get[Option[String]]("phoneNumber")
    } yield User(id, email, name, roles, avatarUrl, phoneNumber)
  }

  implicit val decodeUsers: Decoder[Users] =
    Decoder[List[User]].at("data").map(Users)

}

The idea is to compose the Decoder of a User, and the Decoder for a collection of Users separately. And then by mapping Users to the Decoder, we wrap the results of the Decoder into the Users case class.

score 1 · Answer 2 · answered Aug 06 '20 at 12:15

If you want to decode List[User] you'd need to create decoder for exact type List[User]. It might look like:

implicit val userDecoder: Decoder[List[User]] = (hCursor: HCursor) => {
    Either.fromOption(
      hCursor.downField("data").values,
      DecodingFailure("Can't decode data", Nil)
    ).flatMap { values =>
      values.toList.map(_.hcursor).traverse {
        userCursor =>
          for {
            id <- userCursor.get[String]("id")
            email <- userCursor.get[String]("email")
            name <- userCursor.get[String]("name")
            roles <- userCursor.get[List[String]]("roles")
            avatarUrl <- userCursor.get[Option[String]]("avatarUrl")
            phoneNumber <- userCursor.get[Option[String]]("phoneNumber")
          } yield User(id, email, name, roles, avatarUrl, phoneNumber)
      }
    }
  }

Then you can use it like:

json.as[List[User]]

This might be not the best idea, since circe can already decode json arrays into list, so it might be better idea, to just create decoder for user (decoding object):

implicit val userDecoder: Decoder[User] = (hCursor: HCursor) =>
  for {
    id <- hCursor.get[String]("id")
    email <- hCursor.get[String]("email")
    name <- hCursor.get[String]("name")
    roles <- hCursor.get[List[String]]("roles")
    avatarUrl <- hCursor.get[Option[String]]("avatarUrl")
    phoneNumber <- hCursor.get[Option[String]]("phoneNumber")
} yield User(id, email, name, roles, avatarUrl, phoneNumber)

In this case you'd have manually go down data field:

json.hcursor.downField("data").as[List[User]]

Yet another possibility is to just create case class for data wrapper:

  case class Data[D](data: D)

  object Data {
    implicit def dataDecoder[D: Decoder]: Decoder[Data[D]] =
      (hCursor: HCursor) => hCursor.get[D]("data").map(Data(_))
  }

Then you can do something like:

json.as[Data[List[User]]]

How can I deserialize an non-fixed array of jsons using Circe's manual decoder?

2 Answers2