2

I have a StreamSets container in docker compose and jar file, which are created according with tutorial - https://github.com/streamsets/tutorials/blob/master/tutorial-processor/readme.md

https://github.com/streamsets/tutorials/tree/master/tutorial-processor/sampleprocessor

The source was added correctly.

enter image description here

Extract the tarball to the SDC user-libs directory, restart SDC, and you should see the sample stages in the stage library:

To add Custom Processor to pipeline I copied jar file to user-labs And restart docker

docker exec -it docker-hadoop_streamsets_1 bash
sudo cp -a /data/lib/. /opt/streamsets-datacollector-3.9.0/user-libs
docker restart docker-hadoop_streamsets_1

But after restarting I don't see new destination. Also I tried to add with web interface and read this article - https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Installation/AddtionalStageLibs.html#concept_evs_xkm_s5

enter image description here

bash-4.4$ ls -R /opt/streamsets-datacollector-3.9.0/user-libs/
/opt/streamsets-datacollector-3.9.0/user-libs/:
com.example-1.0-SNAPSHOT          com.example-1.0-SNAPSHOT.jar      com.example-1.0-SNAPSHOT~         sampleprocessor                   sampleprocessor-1.0-SNAPSHOT      sampleprocessor-1.0-SNAPSHOT.jar
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~:
CredentialStores.json                     Interceptors.json                         PipelineStages.json                       com                                       default.png
Delegates.json                            LineagePublishers.json                    Services.json                             data-collector-library-bundle.properties
ElDefinitions.json                        META-INF                                  StageDefList.json                         datacollector-resource-bundles.json
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF:
MANIFEST.MF  maven
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF/maven:
com.example
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF/maven/com.example:
com.example
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF/maven/com.example/com.example:
pom.properties  pom.xml
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com:
example
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example:
stage
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage:
destination  lib          origin       processor
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/destination:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/destination/sample:
Groups.class          SampleDTarget.class   SampleTarget$1.class  SampleTarget.class
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/lib:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/lib/sample:
Errors.class
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/origin:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/origin/sample:
Groups.class         SampleDSource.class  SampleSource.class
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/processor:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/processor/sample:
Groups.class            SampleDProcessor.class  SampleProcessor.class
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor:
ElDefinitions.json                        PipelineStages.json                       data-collector-library-bundle.properties  default.png
META-INF                                  com                                       datacollector-resource-bundles.json
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/META-INF:
MANIFEST.MF  maven
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/META-INF/maven:
com.example
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/META-INF/maven/com.example:
sampleprocessor
Vadim
  • 753
  • 8
  • 22

1 Answers1

1

All you need is the com.example-1.0-SNAPSHOT.jar file, and it should be located in /opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/lib.

metadaddy
  • 4,234
  • 1
  • 22
  • 46