2

I've forked Confluent's Kafka Connect HDFS writer and now I'd like to deploy a version of this jar to my local Nexus.

mvn clean deploy Works like a charm and deploys the jar.

https://[nexus]/repository/releases/io/confluent/kafka-connect-hdfs/5.0.0/kafka-connect-hdfs-5.0.0.jar

So far so good, but to make a distinction between the confluent versions and my own deployment I'd like to change the version of the build to something like 5.0.0-1 or so (preferably the tag name when pushed, but that's step 2)

The pom.xml is basically the same as the 5.0.0-post release, but here the most important parts:

    <parent>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-connect-storage-common-parent</artifactId>
        <version>5.0.0</version>
    </parent>

    <artifactId>kafka-connect-hdfs</artifactId>
    <packaging>jar</packaging>
    <name>kafka-connect-hdfs</name>
    <organization>
        <name>Confluent, Inc.</name>
        <url>http://confluent.io</url>
    </organization>
    <url>http://confluent.io</url>
    <description>
        A Kafka Connect HDFS connector for copying data between Kafka and Hadoop HDFS.
    </description>
...
<dependencies>
    ...
    <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-connect-storage-common</artifactId>
        <version>${confluent.version}</version>
    </dependency>
    <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-connect-storage-core</artifactId>
        <version>${confluent.version}</version>
    </dependency>
    ...
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>io.confluent</groupId>
            <version>0.10.0</version>
            <artifactId>kafka-connect-maven-plugin</artifactId>
...

So first I added <version> tags to the pom.xml, but it started using that as the default for all confluent.version and complained that it couldn't find for example: https://[nexus]/repository/releases/io/confluent/kafka-connect-storage-hive/5.0.0-1/kafka-connect-storage-hive-5.0.0-1.pom

Next I tried the versions plugin from maven mvn versions:set -DnewVersion=5.0.0-1 clean deploy

But that complained about the parent:

[ERROR] Failed to execute goal org.codehaus.mojo:versions-maven-plugin:2.7:set (default-cli) on project kafka-connect-hdfs: Project version is inherited from parent. -> [Help 1]

I don't even care if the version is 5.0.0 in the code, I just wat to deploy to a different version in our artifactory.

I'm not a maven expert, so maybe I'm missing some very basic clue, but all help is welcome.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Tom Lous
  • 2,819
  • 2
  • 25
  • 46
  • Where is confluent.version defined? What is its definition? – J Fabian Meier Nov 19 '18 at 13:13
  • I actually don't know. I'm assuming that the `kafka-connect-maven-plugin` is doing some magic in this area, because I cannot find a reference in this project. – Tom Lous Nov 19 '18 at 13:15
  • When I used extensive debugging it all of a sudden has this line `[DEBUG] properties used {docker.upstream-registry=placeholder/, ... confluent.version=5.0.0, avro.version=1.8.1, java.vm.info=mixed mode, ... ` – Tom Lous Nov 19 '18 at 13:17
  • 1
    Personally, I just use our existing Nexus repos, but have `kafka-connect-hdfsprovided5.0.0`, then just extend whatever code I need... Then, if you put that built JAR into the `kafka-connect-storage-hdfs` folder, it should work – OneCricketeer Nov 19 '18 at 16:52

2 Answers2

2

So there were some good suggestions, but in the end the one thing that worked best for me in our setup was to use the deploy:deploy-file command for maven.

mvn deploy:deploy-file \
-Dfile=target/kafka-connect-hdfs-5.0.0.jar \
-DrepositoryId=[nexus id] \
-Durl=[nexus url] \
-Dversion=$TAG \
-DgroupId=io.confluent \
-DartifactId=kafka-connect-hdfs

The major downside was that I had to respecify parameters that were already present in the pom.xml (artifactId, groupId, ect), but it works, and that's what counts :-)

Tom Lous
  • 2,819
  • 2
  • 25
  • 46
1

You can specify the version with ${revision} parameter.

To do this, you need to add <version> tag with this variable in pom.xml:

<artifactId>kafka-connect-hdfs</artifactId>
<version>5.0.0-${revision}</version>
<packaging>jar</packaging>

And then provide it to the maven command. E.g., mvn clean package -Drevision=01 will generate kafka-connect-hdfs-5.0.0-01.jar file.

biruk1230
  • 3,042
  • 4
  • 16
  • 29
  • Thanks biruk1230, I've actually tried this (sorry for not mentioning) but I ran into the same issues – Tom Lous Dec 06 '18 at 10:08
  • That's strange. Did you try to add this code to your pom.xml (under the **** section)? `${project.artifactId}-${project.version}` – biruk1230 Dec 06 '18 at 16:36