Subentity SolrEntityProcessor stops working since SolR 5.x

Question

I use a data import like this

<dataConfig>
    <document name="products">
        <entity
            name="outer"
            dataSource="my_datasource"
            pk="id"
            query="..."
            deltaQuery="..."
            deltaImportQuery="..."
        >
            <entity
                name="solr"
                processor="SolrEntityProcessor"
                url="http://127.0.0.1:8983/solr/${solr.core.name}"
                query="Xid:${outer.Xid}"
                rows="1"
                fl="Id,FieldA,FieldB"
                wt="javabin"
            />
        </entity>
    </document>
</dataConfig>

The interesting part is the sub entity, which uses SolrEntityProcessor. Until (including) SoLR 4.10 everything worked fine, but since 5.1 it doesn't work anymore. It doesn't fail in the meaning, that it tells me, it failed, but it "successfully" stops during the import of the second document.

DIH import one document
DIH fetches the second (outer) entity
DIH stops

In the logs the following Exception appears. It looks like DIH intentionally closes the connection of the SolrEntityProcessor and crashes as soon as it tries to fetch the sub entity for the second document.

java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.IllegalStateException: Connection pool shut down
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)
    at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:444)
    at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:482)
    at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.IllegalStateException: Connection pool shut down
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)
    at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:363)
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:224)
    ... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.IllegalStateException: Connection pool shut down
    at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:62)
    at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:246)
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:514)
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
    ... 5 more
Caused by: java.lang.IllegalStateException: Connection pool shut down
    at org.apache.http.util.Asserts.check(Asserts.java:34)
    at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
    at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217)
    at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
    at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
    at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
    at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
    at org.apache.solr.handler.dataimport.SolrEntityProcessor.doQuery(SolrEntityProcessor.java:198)
    at org.apache.solr.handler.dataimport.SolrEntityProcessor.buildIterator(SolrEntityProcessor.java:147)
    at org.apache.solr.handler.dataimport.SolrEntityProcessor.nextRow(SolrEntityProcessor.java:128)
    at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
    ... 8 more

http://mail-archives.apache.org/mod_mbox/hc-httpclient-users/201202.mbox/%3C1329833515.2916.11.camel@ubuntu%3E and https://github.com/dakrone/clj-http/issues/150 may help you. — Hana Bzh, Jun 23 '15 at 11:38

Praveen Kumar Purushothaman · Accepted Answer · 2015-06-23T11:20:13.013

1

This is a known bug in Solr. Even I have come across! I posted this as an answer because, this is a bug and there's no solution from the author! We have actually downgraded the version in order to get rid of this bug. I am not sure if this helps, but this is what we did at our end.

Update: This may or may not help you, but we resolved the issue by downgrading.

edited Jun 23 '15 at 11:20

answered Jun 23 '15 at 10:48

Praveen Kumar Purushothaman

164,888
24
203
252

I don't like it, but it "helps". Do you know, when this bug was introduced? Which version did you downgrade to? – KingCrunch Jun 23 '15 at 19:08
@KingCrunch Have a look at this: http://stackoverflow.com/questions/15734308/solrentityprocessor-is-called-only-once-for-sub-entities Found something more. – Praveen Kumar Purushothaman Jun 23 '15 at 19:45
The issue tracker is here: https://issues.apache.org/jira/browse/SOLR-3336 But I am unable to open that from my office. We downgraded to Solr `4.3` currently. – Praveen Kumar Purushothaman Jun 23 '15 at 19:46
Sorry to tell you, but thats an entirely different issue. This question was about the SolrEntityProcessor closing during import, but the ticket you linked is about parameter substitution... The only ticket regarding my issue is the one I've created – KingCrunch Jun 23 '15 at 20:19

Subentity SolrEntityProcessor stops working since SolR 5.x

1 Answers1