Solr BlockJoin Indexing for Solr 4.10.1

Question

I am trying to index a nested structure as below and having difficulty indexing both with SOlrJ and the DIH. I have battled with this for a while and would really appreciate some help on this.

How do i fix this with either SolrJ or DIH. Thanks

What i want my data to look like my index:

"docs": [

{
    "name": "MR INCREDIBLE ",
    "id": 101,
    "job": "super hero",
    "_version_": "1483934897344086016"
    "children": [
            {
                "c_name":"Violet"  
                "c_age":10
                "c_gender":"female"
            },
            {
                "c_name":"Dash"  
                "c_age":8
                "c_gender":"male"
            }
    ]
}

]

My schema.xml

<schema name="datasearch" version="1.5">
<uniqueKey>id</uniqueKey>
<fields>
    <field name="_version_" type="long" indexed="true" stored="true" />
    <field name="_root_" type="string" indexed="true" stored="false"/>

    <field name="id" type="string" indexed="true" stored="true" />
    <field name="name" type="text" indexed="true" stored="true" />
    <field name="job" type="string" indexed="true" stored="true"/>

    <!-- I want to add children here -->
    <!-- <field name="children" indexed="true" stored="true"/> -->
    <field name="c_name" type="string" indexed="true" stored="true"/>
    <field name="c_age" type="int" indexed="true" stored="true"/>
    <field name="c_sex" type="string" indexed="true" stored="true"/>
</fields>

<types>
    <fieldType name="string" class="solr.TrieLongField" />
    <fieldType name="int" class="solr.TrieIntField" />
    <fieldType name="date" class="solr.TrieDateField" omitNorms="true" />
    <fieldType name="long" class="solr.StrField" sortMissingLast="true"/>
    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
        <analyzer>
            <tokenizer class="solr.StandardTokenizerFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.PorterStemFilterFactory"/>
        </analyzer>
    </fieldType>
</types>

<defaultSearchField>name</defaultSearchField>

</schema>

SolrJ Attempt

val serverUrl = current.configuration.getString("solr.server.url").get
val solr = new HttpSolrServer(serverUrl)

def testAddChildDoc={
val doc = {
  new SolrInputDocument(){
    addField("id", "101")
    addField("name", "Mr Incredible")
  }
}
val c1 = new SolrInputDocument(){
    addField("c_name", "violet")
    addField("c_age", 10)
}
val c2 = new SolrInputDocument(){
    addField("c_name", "dash")
    addField("c_age", 8)
}

doc.addChildDocument(c1)
doc.addChildDocument(c2)

solr.deleteByQuery("*:*")
solr.add(doc)
solr.commit(true, true)
}

Response

=>ERROR org.apache.solr.core.SolrCore  – org.apache.solr.common.SolrException: [doc=null] missing required field: id
[RemoteSolrException: [doc=null] missing required field: id]

So i go ahead and add id to childDocs making the above

...    
val c1 = new SolrInputDocument(){
    addField("id", "101")
    addField("c_name", "violet")
    addField("c_age", 10)
}
val c2 = new SolrInputDocument(){
    addField("id", "101")
    addField("c_name", "dash")
    addField("c_age", 8)
}
.....

Then rerun the get-all query, now i get the results below

SolrJ Attempt 2 plus get-all query

{
  "responseHeader": {
    "status": 0,
    "QTime": 0,
    "params": {
      "indent": "true",
      "q": "*:*",
      "_": "1415194092582",
      "wt": "json"
    }
  },
  "response": {
    "numFound": 3,
    "start": 0,
    "docs": [
      {
        "id": 101,
        "c_name": violet,
        "c_age": "10",
      },
      {
        "id": 101,
        "c_name": dash,
        "c_age": "8"
      },
      {
        "id": 101,
        "name": "Mr Incredible",
        "_version_": "1483938552238571520"
      }
    ]
  }
}

So i give up here and try the DIH as below

db-dataconfig.xml

<dataConfig>
    <dataSource type="JdbcDataSource"
                driver="org.postgresql.Driver"
                url="jdbc:postgresql://xxx:5432/xxxx"
                user="xx" password="xx"
                readOnly="true" autoCommit="false" transactionIsolation="TRANSACTION_READ_COMMITTED" holdability="CLOSE_CURSORS_AT_COMMIT" />
    <document>
        <entity name="parent" query="select id,name, job from PARENTS LIMIT 1" >
            <field column="name"/>
            <field column="id"/>
            <field column="job"/>

                <entity child="true" name="children" query="select c_name, c_gender, c_age from CHILDREN" where="pid = ${parent.id}" processor="CachedSqlEntityProcessor">
                    <field column="c_age" />
                    <field column="c_gender" />
                    <field column="c_name"/>
                </entity>
        </entity>
    </document>
</dataConfig>

query get-all after full import with DIH as above and no children indexed

{
  "responseHeader": {
    "status": 0,
    "QTime": 0,
    "params": {
      "indent": "true",
      "q": "*:*",
      "_": "1415195060664",
      "wt": "json"
    }
  },
  "response": {
    "numFound": 1,
    "start": 0,
    "docs": [
      {
        "name": "Mr Incredible",
        "id": 101,
        "_version_": "1483939357483073536"
      }
    ]
  }
}

Answer 1

To be able to use child="true" in DIH apply the patch from https://issues.apache.org/jira/browse/SOLR-5147 (I think it's the same DIH patch at solr-3076).

The patch itself seems to be incompatible in neglectable details with the current trunk.

Answer 2

In order to get the following response from Solr 4.10.1

{
    "name": "MR INCREDIBLE ",
    "id": 101,
    "job": "super hero",
    "type": "parent",
    "_root_":"101"
    "_version_": "1483934897344086016"
    "childDocuments": [
        {
            "c_name":"Violet",
            "c_age":10,
            "c_gender":"female",
            "id":"101_Violet",
            "_root_":"101"
        },
        {
            "c_name":"Dash",
            "c_age":8,
            "c_gender":"male",
            "id":"101Dash",
            "_root_":"101"
        }
    ]
}

"type" field needs to be defined in the schema to differentiate between parent and child documents:

<fields>
    <field name="_version_" type="long" indexed="true" stored="true" />
    <field name="_root_" type="string" indexed="true" stored="false"/>

    <field name="id" type="string" indexed="true" stored="true" />
    <field name="name" type="text" indexed="true" stored="true" />
    <field name="job" type="string" indexed="true" stored="true"/>

    <field name="c_name" type="string" indexed="true" stored="true"/>
    <field name="c_age" type="int" indexed="true" stored="true"/>
    <field name="c_gender" type="string" indexed="true" stored="true"/>

    <field name="type" type="string" indexed="true" stored="true" />
</fields>

Child documents also need to have an unique "id", just like any other document. All the documents in the index should be in parent/child relation, otherwise the queries may return unexpected results. In case you need documents which are neither parents or children, assign them a fake parent.

SolrJ

To work with child/parent docs, solrj.jar version 4.5 or higher is required.

SolrServer solr = new HttpSolrServer(serverUrl);

SolrInputDocument doc = new SolrInputDocument();
String id = "101";
doc.addField("id", id);
doc.addField("name", "Mr Incredible");
doc.addField("job", "super hero");
doc.addField("type", "parent");

SolrInputDocument childDoc1 = new SolrInputDocument();
String name1 = "Violet";
childDoc1.addField("id", id + "_" + name1);
childDoc1.addField("c_name", name1);
childDoc1.addField("c_age", 10);
childDoc1.addField("c_gender", "female");
doc.addChildDocument(childDoc1);

SolrInputDocument childDoc2 = new SolrInputDocument();
String name2 = "Dash";
childDoc2.addField("id", id + "_" + name2);
childDoc2.addField("c_name", name2);
childDoc2.addField("c_age", 8);
childDoc2.addField("c_gender", "male");
doc.addChildDocument(childDoc2);

solr.add(doc);
solr.commit();

Finally, the query looks like this:

http://localhost/solr/core/select?q={!parent which='type:parent'}&fl=*,[child parentFilter=type:parent]&wt=json&indent=true

To get only results of female gender:

http://localhost/solr/core/select?q={!parent which='type:parent'}c_gender:female&fl=*,[child parentFilter=type:parent childFilter=c_gender:female]&wt=json&indent=true