I am using streamsets on CDH version 5.7.0
A sample workflow to load a file from HDFS(origin) and create records on Solr (destination). It is failing on validation - SOLR_03 - Could not connect to the Solr instance: org.apache.solr.common.cloud.ZooKeeperException: : ZooKeeper Connection String.
However, from the logs, I see that it is able to connect to zookeeper, but, somehow closes the session after calling the /live_nodes zookeeper api.
2016-06-03 16:30:21,336 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-06-03 16:30:21,336 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkACLProvider
2016-06-03 16:30:21,338 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 1,3 replyHeader:: 1,42949673777,0 request:: '/clusterstate.json,F response:: s{38654846788,38654846788,1464989414801,1464989414801,0,0,0,0,0,0,38654846788}
2016-06-03 16:30:21,339 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 2,3 replyHeader:: 2,42949673777,0 request:: '/aliases.json,F response:: s{38654846789,38654846789,1464989414814,1464989414814,0,0,0,0,0,0,38654846789}
2016-06-03 16:30:21,339 INFO org.apache.solr.common.cloud.ZkStateReader: Updating cluster state from ZooKeeper...
2016-06-03 16:30:21,339 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 3,3 replyHeader:: 3,42949673777,0 request:: '/clusterstate.json,T response:: s{38654846788,38654846788,1464989414801,1464989414801,0,0,0,0,0,0,38654846788}
2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 4,8 replyHeader:: 4,42949673777,-101 request:: '/live_nodes,T response:: v{}
2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ZooKeeper: Closing session: 0x355187691370039
2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ClientCnxn: Closing client for session: 0x355187691370039
2016-06-03 16:30:21,352 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,42949673778,0 request:: null response:: null
2016-06-03 16:30:21,352 DEBUG org.apache.zookeeper.ClientCnxn: Disconnecting client for session: 0x355187691370039
2016-06-03 16:30:21,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x355187691370039 closed
2016-06-03 16:30:21,352 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-06-03 16:30:21,352 INFO com.streamsets.pipeline.cluster.ControlChannel: Producer complete
2016-06-03 16:30:21,352 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returning private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib'
2016-06-03 16:30:21,352 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returned a private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib', active private ClassLoaders='0'
2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager is shutting down
2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager shut down
2016-06-03 16:30:21,353 DEBUG com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager: Status of previewer with id: '0bec83af-c152-4519-98f4-201394bebc36' changed to status: 'INVALID'
2016-06-03 16:30:21,353 INFO com.streamsets.pipeline.cluster.ControlChannel: Producer complete
2016-06-03 16:30:21,353 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returning private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib'
2016-06-03 16:30:21,353 WARN com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Could not return a private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib', active private ClassLoaders='0'
2016-06-03 16:30:21,353 WARN com.streamsets.datacollector.runner.Pipeline: Exception thrown during pipe 'StagePipe[instance='com_streamsets_pipeline_stage_origin_hdfs_cluster_ClusterHdfsDSource_01' input='[]' output='[com_streamsets_pipeline_stage_origin_hdfs_cluster_ClusterHdfsDSource_01OutputLane14649766295360::s]']' destroy: java.lang.RuntimeException: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid
java.lang.RuntimeException: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid
at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.releaseStageClassLoader(ClassLoaderStageLibraryTask.java:448)
at com.streamsets.datacollector.runner.preview.PreviewStageLibraryTask.releaseStageClassLoader(PreviewStageLibraryTask.java:92)
at com.streamsets.datacollector.creation.StageBean.releaseClassLoader(StageBean.java:60)
at com.streamsets.datacollector.runner.StageRuntime.destroy(StageRuntime.java:203)
at com.streamsets.datacollector.runner.StagePipe.destroy(StagePipe.java:275)
at com.streamsets.datacollector.runner.Pipeline.destroy(Pipeline.java:187)
at com.streamsets.datacollector.runner.preview.PreviewPipeline.destroy(PreviewPipeline.java:71)
at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:205)
at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$2.call(AsyncPreviewer.java:96)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:233)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:484)
at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.releaseStageClassLoader(ClassLoaderStageLibraryTask.java:442)
... 15 more
2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager is shutting down
2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager shut down
So, to verify, I connected to zookeeper and queried the live_nodes:
[root@hadoop3 ~]# zookeeper-client
Connecting to localhost:2181
Welcome to ZooKeeper!
[zk: localhost:2181(CONNECTED) 5] ls /solr
[configs, overseer, aliases.json, live_nodes, collections, overseer_elect, clusterstate.json, solr.xml, zkdtsm, clusterprops.json, token]
[zk: localhost:2181(CONNECTED) 6] ls /solr/live_nodes
[hadoop3:8983_solr, hadoop4:8983_solr, hadoop2:8983_solr, hadoop5:8983_solr]
Not sure, why the response is empty when streamsets calls the API and why it complains about "Zookeeper Connection String" when it is able to connect... Any help?
