5

I would like to be able to do some data manipulation when documents are updated or created in Couchbase.

Documents can arrive in our database either via Sync Gateway or our own code which streams data in from an http service. It would be great to have one place where I can intercept all updates.

We are running a Spring Boot REST API against this data so this would be the good place to have the interceptor/listener. Either way my preference would be for a Java solution.

The data is written as JSON rather than using Spring entities so I can't use ApplicationListener which only listens to events on Entity classes. Correct me if I'm wrong. I can find precious few examples of setting up ApplicationListeners so I may be wrong here but I can't seem to get it working.

I see that there is an Eventing service where you write Javascript but for a number of reasons I'm not keen to go that way. I'm not keen on fragmenting our API code across platforms and languages, not sure I can run the eventing service on our systems etc. Again, I'm open to debate though.

That leaves DCP only as far as I can tell which seems very low level. https://blog.couchbase.com/couchbases-history-everything-dcp/ but looks like the tool for the job.

The QUESTION: Is there an alternative, less low level, way to catch update events in Couchbase for JSON objects NOT entities other than DCP.

Simbosan
  • 244
  • 2
  • 11
  • I'm just curious why would you choose Couchbase if it does not fully match your requirements? Asking because the project I'm currently in also went the same way. – yegodm Oct 17 '18 at 01:17
  • Thanks for taking the time to read the Q. Couchbase is a given because Sync Gateway and Android are involved. – Simbosan Oct 17 '18 at 02:18
  • Some great information here. I followed a different path and used a Kafka Connector to stream events from the Couchbase server into a Kafka topic which I can then forward to the relevant services. This is a very extensible and so far very reliable solution for us, but obviously depends on a Kafka backbone. Just FYI for anyone reading this. The Couchbase connector we use is the Debezium connector and works great without modification – Simbosan Jul 07 '20 at 23:45

2 Answers2

5

Disclaimer: I work for Couchbase and develop the Java DCP client.

If you've already evaluated the Eventing service and decided it doesn't meet your requirements, the Java DCP client might be worth looking into even though it's not officially supported. It's used by the official Couchbase connectors for Kafka, Spark, and Elasticsearch (all of which are open source) and is actively maintained.

If you only care about events that happened since your app started up, usage can be as simple as registering a callback and starting the event stream. Things get a bit more complicated if you need to remember your place in the stream and resume later (to process events that occurred while you were offline, for example), but there's example code for that case too.

The DCP protocol itself is well documented. If you decide to go this route, it might be good to read at least the Architecture section of that documentation. Also be aware that because the Java DCP Client is unsupported, the API can change without notice. (Officially supporting the library and providing a friendlier API are among our long-term goals, but we haven't committed to anything yet.)

Cosimo
  • 2,846
  • 1
  • 24
  • 26
dnault
  • 8,340
  • 1
  • 34
  • 53
  • Thanks much, the code to catch offline events is particularly interesting, I had missed that. I saw the other connectors used DCP which gives me enough confidence, and it seems well documented as you say. – Simbosan Oct 17 '18 at 02:20
2

Like David, I also work for Couchbase as a product manager for the Eventing service.

I would like to be able to do some data manipulation when documents are updated or created in Couchbase.

Eventing certainly allows anyone to respond to and perform data manipulation on mutations (inserts or upserts) via tiny JavaScript fragments. Just take a look at couchbase-eventing-small-scripts-that-solve-big-problems for a quick introduction and also the eventing-examples from the documentation.

If you do go the Eventing service route on a SGW enabled bucket you will need to suppress a duplicate mutation via the crc64() function built into Eventing (for details goto eventing-language-constructs and search for: Sync Gateway). In addition if you want to have Eventing directly update the source bucket if SGW is enabled on that bucket there is a more involved workaround (just reach out to me and I will be happy to provide it)

Next you stated:

not sure I can run the Eventing service on our systems

The Eventing service bundled with the Couchbase Enterprise offering, it provides scalable infrastructure to run simple JavaScript fragments on data or documents as they change or mutate without the overhead of an SDK. You either add stand alone Eventing node(s) to your Couchbase cluster or collocate the Eventing service with other existing nodes.

Jon Strabala
  • 421
  • 3
  • 3