3

Disclaimer: I am new to sttp and Monix, and that is my attempt to learn more about these libraries. My goal is to fetch data (client-side) from a given API via HTTP GET requests -> parse JSON responses -> write this information to a database. My question pertains to the first part only. My objective is to run get requests in an asynchronous (hopefully fast) way while having a way to either avoid or handle rate limits.

Below is a snippet of what I have already tried, and seems to work for a single request:

package com.github.client

import io.circe.{Decoder, HCursor}
import sttp.client._
import sttp.client.circe._
import sttp.client.asynchttpclient.monix._
import monix.eval.Task

object SO extends App {

  case class Bla(paging: Int)

  implicit val dataDecoder: Decoder[Bla] = (hCursor: HCursor) => {
    for {
      next_page <- hCursor.downField("foo").downArray.downField("bar").as[Int]
    } yield Bla(next_page)
  }

  val postTask = AsyncHttpClientMonixBackend().flatMap { implicit backend =>
    val r = basicRequest
      .get(uri"https://foo.bar.io/v1/baz")
      .header("accept", "application/json")
      .header("Authorization", "hushh!")
      .response(asJson[Bla])

    r.send() // How can I instead of operating on a single request, operate on multiple
      .flatMap { response =>
        Task(response.body)
      }
      .guarantee(backend.close())
  } 

  import monix.execution.Scheduler.Implicits.global

  postTask.runSyncUnsafe() match {
    case Left(error) => println(s"Error when executing request: $error")
    case Right(data) => println(data)
  }
}

My questions:

  1. How can I operate on several GET Requests (instead of a single request) via using Monix, while keeping the code asynchronous and composable
  2. How can I avoid or handle hitting rate-limits imposed by the api server

On a side note, I am also flexible in terms of using another back-end if that will support the rate limiting objective.

alt-f4
  • 2,112
  • 17
  • 49

1 Answers1

2

You can use monix.reactive.Observable like this

  Observable.repeatEval(postTask) // we generate infinite observable of the task
    .throttle(1.second, 3) // set throttling
    .mapParallelOrderedF(2)(_.runToFuture) // set execution parallelism and execute tasks
    .subscribe() // start the pipline
  
  
  while (true) {}
Artem Sokolov
  • 810
  • 4
  • 8
  • Hi Artem, welcome to stack overflow. Although your answer did not fully solve my issue, it was indeed helpful for getting closer to the solution : ) I have further questions on the approach, but I decided it is better to cover in a separate question here: https://stackoverflow.com/questions/63318135/how-can-i-throttle-sending-http-get-requests-via-monix – alt-f4 Aug 08 '20 at 17:11