0

I am new to PouchDB and I am trying to simply pull data from my remote CouchDB database to a new local PouchDB database.

The following works of course:

PouchDB.replicate(remote, local, { live: false, retry: false })

but it is just crazy how slow it is! I take it is because it's doing so many http requests sequentially.

As an optimisation I thought:

OK, let's fetch the data first in one HTTP request with remote.allDocs then use local.bulkDocs to insert the docs in the local db

but it's not equivalent to replicating the db, and the _rev field in the docs will make bulkDocs fail.

What should I do?

More details

about the database

  • there currently are 1202 documents
  • there use to be about 4000
  • I just added a couple of revisions to each document

about the performances:

  • it takes about 2 seconds to run remote.allDocs({include_docs: true}). The output is 1.15 MB
  • it takes 1 minute to run PouchDB.replicate(remote, local, { live: false, retry: false }), the output is about 5 MB
  • it takes about 24 seconds to add a filter: doc => !doc._deleted
  • and about 20 seconds to add a filter: 'new_device/excludeDeletedDocs' with the same filter as before in a design document

About the replication lasting 1 minute: it surprised me, I thought it was faster than that, but it's how long it takes now.

about the app

  • there is one database per user
  • they can access it through multiple devices, online and offline
  • most docs are small, are edited a couple of times and are relatively short lived
  • a big minority of them last virtually forever
  • a fraction of them can be bigger and can be edited many many times.
geoffrey
  • 2,080
  • 9
  • 13
  • You may find some relief with [PouchDB Replication Streams](https://github.com/pouchdb-community/pouchdb-replication-stream), see case #2 [here](https://cnpmjs.org/package/pouchdb-replication-stream). Also [here's a way to preload a database](https://stackoverflow.com/questions/68105040/how-can-i-copy-pouchdb-0000003-log-file-to-ionic-5-and-retrieve-the-data/68245289#68245289) for hybrid apps – RamblinRose Sep 28 '21 at 20:40
  • 1
    As an afterthought, how many revisions is the db retaining? Often lowering that threshold to a reasonable level will have a noticeable impact for some databases. See [revs_limit](https://pouchdb.com/api.html#create_database). Without specific information regarding your database you may not get a definitive answer. – RamblinRose Sep 28 '21 at 20:43
  • Hi, Thanks I'll investigate that tomorrow morning. I have about 1000 docs in my database, they haven't been updated once since I'm importing old data from a previous version of the app, but I've played around with the remote DB, there use to be 4000 items, the there are tombstones indeed. However I expect that a normal db in a real app would have a certain number of revisions anyway. – geoffrey Sep 28 '21 at 20:54
  • indeed - but a 1000 docs is trivial, unless there is an endless stream of revs especially if the docs are largish. Follow up with an edit of your OP with more info! – RamblinRose Sep 28 '21 at 23:25
  • At least sounds like you want to at least use [options.filter](https://pouchdb.com/api.html#replication) to purge deleted docs. – RamblinRose Sep 28 '21 at 23:35
  • "1.15Mo ... 5 Mo" what is "Mo"? – RamblinRose Sep 29 '21 at 13:09
  • 1
    Sorry, this is a French designation. A megaoctet is just a megabyte (with one byte being 8 bits which is the case most of the time). Anyway, it's a vert small db for the time it takes to sync it. – geoffrey Sep 29 '21 at 15:55
  • Is it true that the initial database which you are replicating to a client device is fairly static, in other words is it mainly a starting point? – RamblinRose Sep 29 '21 at 17:42
  • The initial database is representative of data that can be produced by one user during one year. The amount of data should grow at a decreasing rate, then linearly. I expect that at some point some databases will have multiple users with more frequent edits, but nothing crazy. That being said, even a single user can manage to create conflicts when on multiple devices which can go offline. – geoffrey Sep 29 '21 at 19:02
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/237653/discussion-between-ramblinrose-and-geoffrey). – RamblinRose Sep 29 '21 at 20:57

0 Answers0