1

I'm currently trying to find a good way to notify me if some documents in my index have changed. Now some questions came up, and I hope someone can help me here. I'm using Solr 6.2.1.

The SolrEventListener Interface has the methods postCommit() and postSoftCommit(), but in the documentation (http://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-6.2.pdf) there are only informations about the two events "postCommit" and "postOptimize".

How are these related? I only managed to get triggered postCommit() on "postCommit", but I would like to get a notification on every soft-commit. I figured that is the postSoftCommit() method for, but i could not find any event to trigger a call to this method. And just for curiosity, what triggers "postOptimize"?

Also, am I right with the assumption that even, if i set autoCommit and autoSoftCommit, they only get triggered if there are actual changes?

Is there a possibility to find out how many documents have changed since the last commit?

I'm of course also open to other suggestions how to achieve this behavior.

EDIT for clarity:

If i add the fallowing to my solrconfig.xml

<listener event="postCommit" class="com.solr.plugins.event.NotifyChangesEventListener"/>

com.solr.plugins.event.NotifyChangesEventListener.postCommit() gets called as expected.

But what is the right event, so that com.solr.plugins.event.NotifyChangesEventListener.postSoftCommit() gets called?

I tried 

<listener event="postSoftCommit" class="com.solr.plugins.event.NotifyChangesEventListener"/>

but it did not have any effects.

And what method gets called if I add 

<listener event="postOptimize" class="com.solr.plugins.event.NotifyChangesEventListener"/>
Mathias Mahlknecht
  • 281
  • 2
  • 6
  • 16

2 Answers2

2

Updates after the question was updated:

I think you're correct - it seems the postSoftCommit event is only exposed to plugins that directly subscribe by calling registerSoftCommitCallback instead of through solrconfig, see the code that handles registration from solrconfig. postOptimize calls the same method as postCommit, so the interface only defines postCommit and calls it both on optimize and on commit.

To register for the softCommitCallback, your plugin should be SolrCoreAware and it should receive a call to inform when a core is registered, where you can call registerSoftCommitCallback. I guess (I'm not familiar with the code or have submitted a patch for this part before) a patch that adds softCommitCallback handling in solrconfig.xml would be merged, as it seems like an oversight (no comments that say otherwise at least that I've found).

Each of the post*() methods corresponds to an update action taken by a client for the index.

  • A soft commit is a wish to make documents visible for the searcher, but not necessarily persisted to disk before this happens. It's more important to get stuff visible instead of waiting for stuff to be written to disk. This is useful for a busy index that's being updated often, but instead of having to write and persist the index a couple of times every second, just make the changes visible. The added document is usually persisted to the transaction log as well in a cluster environment, so all hope is not lost even if stuff crashes.

  • A hard commit (or just a commit) persists the changes to disk.

  • An optimize rewrites all the segment files as a single, large segment. This is usually not required since Lucene will merge segments as the number of segments gets larger. But if you generate your index once each night, running optimize and then distributing that index will give you a more performant index and a smaller index size (since no old, deleted documents will be lingering around on disk).

I'm not sure which documentation you're referring to, but all the post*() handlers are called from the DirectUpdateHandler (Line 705).

Yes, the autocommits should only get triggered if there are documents in the queue. No need to commit non-existant data.

You can perform a softCommit explicitly by having softCommit=true set in your commit request.

I don't think you can get the number of added documents from the event (.. but look at how the log entry is implemented in that case, since it shows the ids added), but since you receive the searchers, you might be able to get the reader from the searcher and query it for numDocs. If you keep the last count around in your listener, you should be able to determine the delta count between events.

MatsLindh
  • 49,529
  • 4
  • 53
  • 84
  • Ok, thanks for your answers. But i think you miss understood my main question. The difference between soft and hard commits are clear to me, I just wanted to know how I can get solr to trigger the different events. I edited my question, maybe now it is more clear. – Mathias Mahlknecht Sep 04 '17 at 14:41
  • So are you actually doing softcommits and optimizes? And you're not seeing your events being called? – MatsLindh Sep 04 '17 at 14:44
  • Yes and I think it is because I'm not defining the right "event", because if I set event="postCommit" I see "o.a.s.u.UpdateHandler added SolrEventListener for postCommit: com.solr.plugins.event.NotifyChangesEventListener{}" in the logs on startup, but for "postSoftCommit" and "postOptimize" I don't find any initialization in the logs. Regarding "postOptimize" I'm not sure which methods should be called, because there is no corresponding method in the Interface. – Mathias Mahlknecht Sep 04 '17 at 14:51
  • I think you're correct - it seems the `postSoftCommit` event is only exposed to plugins that directly subscribe instead of through solrconfig, [see the code that handles registration from solrconfig](https://github.com/apache/lucene-solr/blob/5d9b9d7a23447c810d53a85c63eab122099b2d2f/solr/core/src/java/org/apache/solr/update/UpdateHandler.java#L67). `postOptimize` calls the same method as `postCommit`. They're analogous in what they mean for external code, apparently, just that you're able to attach to the event explicitly if you want it. – MatsLindh Sep 04 '17 at 14:59
  • Ok I see. Maybe this possibility should be added in the future. Do you know a way how I can subscribe to this event directly? Or is it just not possible for users and I have to stick with less "real time" notifications throw postCommit() ? – Mathias Mahlknecht Sep 04 '17 at 15:08
  • Your plugin must be [SolrCoreAware](https://wiki.apache.org/solr/SolrPlugins#SolrCoreAware), where you can call [registerSoftCommitCallback](https://github.com/apache/lucene-solr/blob/5d9b9d7a23447c810d53a85c63eab122099b2d2f/solr/core/src/java/org/apache/solr/update/UpdateHandler.java#L203) from your `inform` method. I guess (I'm not familiar with the code or have submitted a patch for this part before) a patch that adds softCommitCallback handling in solrconfig.xml would be merged, as it seems like an oversight (no comments that say otherwise at least that I've found). – MatsLindh Sep 04 '17 at 18:22
  • 1
    Thanks, you already helped me a lot. I marked your answer as accepted, but I have one more question. Is there a special way to register a SolrCoreAware plugin? Because if I just import it with , inform() is never called. I also tried to register it for hard commits with , then it gets registered for hard commits, but inform() is still never called. – Mathias Mahlknecht Sep 05 '17 at 09:15
  • 1
    Ok, I upgraded now to version 6.6.0. Here inform() is called for all registered Listeners, so I finally got it to work. Thanks again. – Mathias Mahlknecht Sep 05 '17 at 09:29
0

Since you said you were open to alternative solutions...

As of Solr 7.0, you can subscribe to a query

mancini0
  • 4,285
  • 1
  • 29
  • 31