Solr DataImportHandler configuration

Question

I want to get data from mysql database with the help of DataImportHandler so i can create indexes. Now I've configured my Solr instance so that it works on Tomcat (the example admin page), but if I try to change the sorlconfig.xml file i'll get the error message. I'm working with Solr 3.6

So my configuration is:

In solrconfig.xml I added:

<dataDir>${solr.data.dir:/usr/share/tomcat7/solr2}</dataDir>

to specify my working directory

and then

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">/usr/share/tomcat7/solr2/conf/data-config.xml</str> 
</lst>
</requestHandler>

to specify new request handler.

Theese two are my lib directives for DIH. Do i need to change them?

<lib dir="../../dist/" regex="apache-solr-dataimporthandler-\d.*\.jar" />
<lib dir="../../contrib/dataimporthandler/lib/" regex=".*\.jar" />

I also created data-config.xml file and added following:

<?xml version="1.0" encoding="UTF-8"?>
<dataConfig>
<dataSource   type="JdbcDataSource"
              driver="com.mysql.jdbc.Driver"
              url="jdbc:mysql://localhost:3306/ethicsweb_experts"
              user="root"
              password=""/>

<document>
       <entity name="experts" datasource="mysql" pk="mainid"
              query="SELECT experts.mainid as mainid FROM experts WHERE validRec = 'y'">

        <field column="mainid" name="mainid"/>
        <field column="validRec" name="validRec"/>
    </entity>
</document>

I've coppied following jars to tomcat/lib folder (DIH jar files and mysql JDBC connector jar file)

apache-solr-dataimporthandler-3.6.0.jar
apache-solr-dataimporthandler-extras-3.6.0.jar
mysql-connector-java-5.1.20-bin.jar

Also in the schema.xml file i added folowing fields:

<field name="mainid" type="int" indexed="true" stored="true"  />
<field name="validRec" type="string" indexed="true" stored="true" />
<field name="recSource" type="string" indexed="true" stored="true" />

<uniqueKey>mainid</uniqueKey>

But now when i try to acces:

http://localhost:8080/solr2/

I get following output:

HTTP Status 500 - Severe errors in solr configuration. Check your log files for more detailed information on what may be wrong. If you want solr to continue after configuration errors, change: <abortOnConfigurationError>false</abortOnConfigurationError> in solr.xml ------------------------------------------------------------- 
org.apache.solr.common.SolrException: No cores were created, please check the logs for errors at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:172) 
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:96) 
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277) 
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258)
at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382) 
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103) 
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638) 
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) 
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895) 
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871) 
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615) 
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649) 
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1585) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 
at java.util.concurrent.FutureTask.run(FutureTask.java:166) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) 
at java.lang.Thread.run(Thread.java:722)
 -------------------------------------------------------------         
java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware 
at java.lang.ClassLoader.defineClass1(Native Method) 
at java.lang.ClassLoader.defineClass(ClassLoader.java:791) 
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) 
at java.net.URLClassLoader.access$100(URLClassLoader.java:71) 
at java.net.URLClassLoader$1.run(URLClassLoader.java:361) 
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
at java.security.AccessController.doPrivileged(Native Method) 
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:423) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:356) 
at java.lang.Class.forName0(Native Method) 
at java.lang.Class.forName(Class.java:264) 
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1698) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:410) 
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:356) 
at java.lang.Class.forName0(Native Method) 
at java.lang.Class.forName(Class.java:264) 
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:378) 
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:419) 
at org.apache.solr.core.SolrCore.createRequestHandler(SolrCore.java:455) 
at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:159) 
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:563) 
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:483) 
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:335) 
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:219) 
at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:161) 
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:96) 
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277) 
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258) 
at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382) 
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103) 
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638) 
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294) 
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) 
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895) 
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871) 
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615) 
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649) 
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1585) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 
at java.util.concurrent.FutureTask.run(FutureTask.java:166) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) 
at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware 
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
at java.security.AccessController.doPrivileged(Native Method) 
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:423) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:356) ... 47 more

My log entries show me that:

SEVERE: java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1698)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
    at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:378)
    at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:419)
    at org.apache.solr.core.SolrCore.createRequestHandler(SolrCore.java:455)
    at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:159)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:563)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:483)
    at org.apache.solr.core.CoreContainer.load(CoreContainer.java:335)
    at org.apache.solr.core.CoreContainer.load(CoreContainer.java:219)
    at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:161)
    at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:96)
    at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277)
    at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258)
    at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382)
    at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103)
    at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1585)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    ... 47 more

Jun 15, 2012 4:07:50 PM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start Solr. Check solr/home property and the logs
org.apache.solr.common.SolrException: No cores were created, please check the logs for errors
    at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:172)
    at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:96)
    at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277)
    at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258)
    at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382)
    at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103)
    at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1585)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Jun 15, 2012 4:07:50 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: No cores were created, please check the logs for errors
    at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:172)
    at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:96)
    at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277)
    at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258)
    at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382)
    at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103)
    at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1585)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

So now I wonder where did I screw up the configuration for the DataImportHandler. Do I need to specify more jar files? Did I put jar files in the correct directory? Any help would be greatly appreciated.

Take a look at [this](http://www.mail-archive.com/solr-user@lucene.apache.org/msg65390.html) comment. — Marko Bonaci, Jun 19 '12 at 09:45
"Did you make sure to delete the files expanded from the previous .war file from your servlet container (most likely either jetty or tomcat) before starting it back up?" But where can i find these files. I'm total newbie with Solr so please if someone knows hot to make DataImportHandler to work with mysql database please help. — talo, Jun 19 '12 at 11:17
That sentence means that you need to examine webapps folder under your Application server and delete extracted solr files. E.g. Tomcat and Jetty both have Server_Root\webapps folder. Remove previous solr's war file and content extracted from that war file before deploying (placing in that folder) a new war. Your error seems to be popping out because your war (or classpath) file does not contain SolrCoreAware class. — Marko Bonaci, Jun 19 '12 at 14:34

score 2 · Accepted Answer · answered Jun 19 '12 at 13:54

2

The issue here is your "lib" entries with jar file location is not properly defined.

Depending on your solr.home entry, these dist, contrib folders are not resolving. For instance, you may need to additional "../" for this..

answered Jun 19 '12 at 13:54

user1452132

1,758
11
21

Tried several different lib directives. But still got errors. Where do i need to put my jar files? And how would the lib directive look? – talo Jun 20 '12 at 09:54
@talo did you resolved that issue? Am facing the same issue and my solr version is 7.4.0 – AshrithGande Jul 25 '18 at 10:07

score 0 · Answer 2 · answered Sep 26 '12 at 13:15

OK I've been battling with the same issue for hours. Something the solr 3.4 book didn't cover about the set up of DIH.

Leave the lib folder as is, in the example folder. Create a new folder inside example/solr called lib. Open up example/solr/solr.xml where the cores are configured and add the sharedLib attribute

<solr persistent="false" sharedLib="./lib">

Dump all your solr jars here! There is some conflict issue with solr not loading them from the lib folder, I think thats for jetty libs.... but I wouldn't know for sure.

score 0 · Answer 3 · edited May 31 '18 at 18:34

0

The war file which your server extracted doesn't have these DIH drivers in lib directory. Add it in there and that might solve your problem.

I was fighting for this for 5 hours and now it got working.

edited May 31 '18 at 18:34

Kraagenskul

424
1
7
18

answered Sep 24 '13 at 16:48

kunal badgujar

1

Solr DataImportHandler configuration

3 Answers3

Linked