1

I have solr 3.6.2 installed on my machine, perfectly running with tomcat. I want to index a wikipedia dump file using solr. How do I do this using DataImportHandler? Any other way? I don't have any knowledge of xml.

The file I have mentioned has size of around 45GB when extracted. Any help would be greatly appreciated.

Update- I tried doing whats said on the DataImportHandler page. But there is some error maybe because their version of solr is much older.

My data.config-

<dataConfig>
    <dataSource type="FileDataSource" encoding="UTF-8" />
    <document>
    <entity name="page"
            processor="XPathEntityProcessor"
            stream="true"
            forEach="/mediawiki/page/"
            url="./data/enwiki.xml"
            transformer="RegexTransformer,DateFormatTransformer"
            >
        <field column="id"        xpath="/mediawiki/page/id" />
        <field column="title"     xpath="/mediawiki/page/title" />
        <field column="revision"  xpath="/mediawiki/page/revision/id" />
        <field column="user"      xpath="/mediawiki/page/revision/contributor/username" />
        <field column="userId"    xpath="/mediawiki/page/revision/contributor/id" />
        <field column="text"      xpath="/mediawiki/page/revision/text" />
        <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />
        <field column="$skipDoc"  regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/>
   </entity>
    </document>

Schema (I just added the parts they have given on the website to my schema.xml file)

The error I am getting is - 

<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
</lst>
<lst name="initArgs">
<lst name="defaults">
<str name="config">solr-data-config.xml</str>
</lst>
</lst>
<str name="command">full-import</str>
<str name="status">idle</str>
<str name="importResponse"/>
<lst name="statusMessages">
<str name="Time Elapsed">0:0:1.381</str>
<str name="Total Requests made to DataSource">0</str>
<str name="Total Rows Fetched">0</str>
<str name="Total Documents Processed">0</str>
<str name="Total Documents Skipped">0</str>
<str name="">Indexing failed. Rolled back all changes.</str>
<str name="Rolledback">2013-05-17 16:48:32</str>
</lst>
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>

please help

srjoglekar
  • 31
  • 4
  • I solved the same issue in this link http://stackoverflow.com/questions/20473798/indexing-wikipedia-with-solr. I hope it helps. – Marcelo Jan 06 '14 at 08:41

1 Answers1

2

Simple post is not the right way to index Wikipedia. You need to look into using DataImportHandler instead. DIH supports streaming import.

Alexandre Rafalovitch
  • 9,709
  • 1
  • 24
  • 27
  • Ya. I got that part after doing some research. However, I am still confused about the usage of DataImportHandler. Could you give me a few starters please? – srjoglekar May 17 '13 at 04:38
  • Try going through the example first, step by step. Then, you can ask a specific question that is not already covered by the example. – Alexandre Rafalovitch May 19 '13 at 23:49
  • @alexandre or all-- I am trying to index wikipedia..but it is not indexing all filelds specified. Please help me for the same-- http://stackoverflow.com/questions/32628350/solr-dataimporthandler-is-not-indexing-all-data-defined – Gaurav Pant Sep 18 '15 at 07:27