1

I am trying to pull records from a view which emits in following way

DefaultViewRow{id=0329a6ac-84cb-403e-9d1d, key=[“X”,“Y”,“0329a6ac-84cb-403e-9d1d”,“31816552700”], value=1}

As we have millions of record, we are trying to implement pagination to pull 500 records per page and do some process then get next 500 records i have implemented following code with java client

def cluster = CouchbaseCluster.create(host)
def placesBucket = cluster.openBucket("pass", "pass")
def startKey = JsonArray.from(X, "Y")
def endKey = JsonArray.from(X, "Y", new JsonObject())
hasRow = true
rowPerPage = 500
page = 0
currentStartkey=""
startDocId=""
def viewResult
def counter = 0
while (hasRow) {
    hasRow = false
   def skip = page == 0 ?0: 1
    page = page + 1
     viewResult = placesBucket.query(ViewQuery.from("design", "view")
            .startKey(startKey)
             .endKey(endKey)
            .reduce(false)
            .inclusiveEnd()
            .limit(rowPerPage)
            .stale(Stale.FALSE)
            .skip(skip).startKeyDocId(startDocId)
    )
    def runResult = viewResult.allRows()
    for(ViewRow row: runResult){
        hasRow = true
        println(row)
        counter++
        startDocId = row.document().id()
    }
    println("Page NUMBER "+ page)
}
println("total "+ counter)

Post execution, i am getting few repetitive rows and even though the total records is around 1000 for particular small scenario i get around 3000+ rows in response and it keeps going. can someone please tell me if i am doing something wrong ?? PS: My start key value will be same for each run as i am trying to get each unique doc _id. please help.

deewreck
  • 113
  • 7
  • what version of Couchbase are you using? Is using SQL++ not an option (and why)? – Matthew Groves Aug 08 '22 at 14:53
  • 1
    i am using 7.10.2 version of couchbase. I am using following in a script where couchbase provides few clients to query this nosql db. i am not sure if i can use sql here. – deewreck Aug 08 '22 at 14:58
  • I'm also a little confused... you are saying it's Java, but it's tagged python. I don't think the code example you're posted is Java. Which are you using, Java or Python? – Matthew Groves Aug 08 '22 at 15:04
  • 1
    sorry, i might have added python by mistake. its java. – deewreck Aug 08 '22 at 15:13
  • This doesn't look like either Java or Python to me. My guess is JavaScript. You are going to have a much better chance of getting your actual question answered if you don't start by misleading people – G. Blake Meike Aug 15 '22 at 17:28
  • @G.BlakeMeike My money's on [Groovy](https://groovy-lang.org). Still the Couchbase Java SDK, even if the code calling it is in a different JVM language. – dnault Aug 15 '22 at 20:19
  • [As Mike Reiche mentioned on the Couchbase Forum](https://forums.couchbase.com/t/pagination-on-couchbase-view/34203), the correct way to do this kind of pagination is to update the `startKey` and `startKeyDocId` values after every iteration of the loop, so the next search starts where the previous ended. – dnault Aug 15 '22 at 20:21

0 Answers0