4

I am trying to consume data from this endpoint on lichess.org.

Here is a minimal working example of a React component consuming that stream of data. I'm using a library called can-ndjson-stream.

import ndjsonStream from "can-ndjson-stream"
import { useEffect } from "react"

function App() {
  useEffect(() => {
    fetch("https://lichess.org/api/tv/feed")
      .then(res => ndjsonStream(res.body))
      .then(stream => {
        const streamReader = stream.getReader()
        streamReader.read().then(async res => {
          while (!res || !res.done) {
            res = await streamReader.read()
            console.log(res.value)
          }
        })
      })
      .catch(console.error)
  }, [])
  return <>Lorem Ipsum</>
}

export default App

However, if I try to write this same code and run it in Node like this:

import ndjsonStream from "can-ndjson-stream"
import fetch from "node-fetch"

fetch("https://lichess.org/api/tv/feed")
  .then(res => ndjsonStream(res.body))
  .then(stream => {
    const streamReader = stream.getReader()
    streamReader.read().then(async res => {
      while (!res || !res.done) {
        res = await streamReader.read()
        console.log(res.value)
      }
    })
  })
  .catch(console.error)

I get this error:

ReferenceError: ReadableStream is not defined at ndjsonStream

So it seems like the res from the fetch is null or undefined, but fetching other APIs works fine.

I also tried using axios instead of node-fetch like this:

import ndjsonStream from "can-ndjson-stream"
import axios from "axios"

axios
  .get("https://lichess.org/api/tv/feed")
  .then(res => ndjsonStream(res.data))
  .then(stream => {
    const streamReader = stream.getReader()
    streamReader.read().then(async res => {
      while (!res || !res.done) {
        res = await streamReader.read()
        console.log(res.value)
      }
    })
  })
  .catch(console.error)

But it just hangs and shows no output. Thanks to anyone that could shed some light on this or offer any alternative way to run this in Node.

geekTechnique
  • 850
  • 1
  • 11
  • 38
  • "node-fetch" is not an exact equivalent of the browser "fetch" API, and it shows in this particular case. In the browser API `res.body` is a `ReadableStream`, and that's what `ndjsonStream` expects. In "node-fetch" you get Node's `PassThrough` stream object, which doesn't provide functionality `ndjsonStream` expects. – tromgy Oct 09 '21 at 12:10
  • And in case of "axios", you need to construct the request with `"stream"` as the response type: `axios({ method: 'get', url: 'https://lichess.org/api/tv/feed', responseType: 'stream' })`, but it still won't produce `ReadableStream` that `ndjsonStream` expects, but with axios it will be `IncomingMessage`, another Node stream type. – tromgy Oct 09 '21 at 12:24
  • This might be of some help: https://css-tricks.com/web-streams-everywhere-and-fetch-for-node-js/#fetch-response-as-a-readable-stream – tromgy Oct 09 '21 at 12:31
  • @tromgy Thanks so much for your help. After reading that article, I have decided to not use "can-ndjson-stream" in favor of a much easier to use library, [hyperquest](https://www.npmjs.com/package/hyperquest). All I have to write is a simple line: `hyperquest("https://lichess.org/api/tv/feed").pipe(process.stdout)` – geekTechnique Oct 09 '21 at 22:21

1 Answers1

3

Thanks to the comments from tromgy, I was able to make something that works. I went with the library hyperquest to help handle the request and piping of the stream. I also used the ndjson library.

Here is some working code:

hyperquest("https://lichess.org/api/tv/feed")
    .pipe(ndjson.parse())
    .on("data", console.log)

Note that you can manipulate the objects as they arrive with the second argument of on(), like below.

...
.on("data", (obj) => {
    foo(obj)
})
geekTechnique
  • 850
  • 1
  • 11
  • 38