java.lang.NoClassDefFoundError in Talend Studio

Question

I want install some Google Cloud Java Client Libraries to use it in a tJava component (I don't want use the talend's pre-build component).

I'm trying some test with a job with a tJava component. This is the code that I want to run:

BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

in the advance setting I import the following libraries:

import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;

To setting the following library I did the following step:

I download the last version of the artifact google-cloud-bigquery from https://search.maven.org/search?q=g:com.google.cloud%20AND%20a:google-cloud-bigquery&core=gav
I download the last version of the artifact google-cloud-core from https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core/1.94.0
I put this two jar in the following folder:

-...\TALEND STUDIO\configuration.m2\repository\com\google\cloud\google-cloud-bigquery\1.126.3\google-cloud-bigquery-1.126.3.jar

-...\TALEND STUDIO\configuration.m2\repository\com\google\cloud\google-cloud-core\1.94.0\google-cloud-core-1.94.0
From the navigator tab I open: NAME_PROJECT/poms/pom.xml and I add the dependencies as following:

using the the bottom "Add" this change the pom.xml as following:

...
  <dependencies>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.126.3</version>
    </dependency>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-core</artifactId>
        <version>1.94.0</version>
    </dependency>
  </dependencies>
</project>

Run the job, not compile errors occur but a runtime error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/cloud/bigquery/BigQueryOptions at bce_datahub.bigquery_template_0_1.bigquery_template.tJava_1Process(bigquery_template.java:331) at bce_datahub.bigquery_template_0_1.bigquery_template.runJobInTOS(bigquery_template.java:638) at bce_datahub.bigquery_template_0_1.bigquery_template.main(bigquery_template.java:464) Caused by: java.lang.ClassNotFoundException: com.google.cloud.bigquery.BigQueryOptions at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 3 more

I saw from this thread Why am I getting a NoClassDefFoundError in Java? that the problem is due to the fact that the class code is not found at runtime. I don't know how to put the class (jar) in the runtime path.

Thank you for your help.

EDIT:

I don't know if it is useful but I put all the pom.xml. I hide (with xxx) some part of it for privacy. I think that the list of modules (talend jobs) isn't useful so I delete a part of that list:

<?xml version="1.0"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.example.xxx</groupId>
  <artifactId>code.Master</artifactId>
  <version>7.0.1</version>
  <packaging>pom</packaging>
  <name>xxx Codes Master</name>
  <url>http://www.talend.org/</url>
  <modules>
    <module>code/routines</module>
    <module>jobs/process/xxx</module>
    <module>jobs/process/xxx/xxx/xxx</module>
    <module>...</module>
    <module>jobs/process/xxx/xxx/xxx</module>
    <module>../../REFERENCE/poms</module>
    <module>jobs/process/xxx</module>
  </modules>
  <properties>
    <talend.project.name>xxx</talend.project.name>
    <encoding>UTF-8</encoding>
  </properties>
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <artifactId>maven-clean-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.6.1</version>
          <dependencies>
            <dependency>
              <groupId>org.eclipse.tycho</groupId>
              <artifactId>tycho-compiler-jdt</artifactId>
              <version>1.0.0</version>
            </dependency>
          </dependencies>
          <configuration>
            <source>1.8</source>
            <target>1.8</target>
            <compilerId>jdt</compilerId>
          </configuration>
        </plugin>
        <plugin>
          <artifactId>maven-resources-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-jar-plugin</artifactId>
          <version>3.0.2</version>
          <configuration>
            <archive>
              <addMavenDescriptor>false</addMavenDescriptor>
            </archive>
          </configuration>
        </plugin>
        <plugin>
          <artifactId>maven-assembly-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.20</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
  <profiles>
    <profile>
      <id>ci-builder</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <build>
        <plugins>
          <plugin>
            <groupId>org.talend</groupId>
            <artifactId>ci.builder</artifactId>
            <version>7.0.1</version>
            <executions>
              <execution>
                <phase>generate-sources</phase>
                <goals>
                  <goal>generate</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
  <dependencies>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.126.3</version>
    </dependency>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-core</artifactId>
        <version>1.94.0</version>
    </dependency>
  </dependencies>
</project>

score 1 · Accepted Answer · answered Dec 28 '20 at 12:20

1

The POM files of the jobs are generated by studio actions, so I'd avoid editing them manually.

There are other ways to achieve this, first I'd suggest to take a look at tLibraryLoad component to add dependency jars: https://help.talend.com/r/lw17672ujtsKAM3zY5H2oQ/wcAnfUMxgsm90CCS6~jiZw

If you have more than 2-3 jar files you could also handle the dependencies via Routines, and then set up the routine as a dependency for the job(s). Keep in mind that if your SDK has additional dependency jars then those should be added as well.

answered Dec 28 '20 at 12:20

Balazs Gunics

2,017
2
17
24

1

Thank you very much. Under the Repository -> Code -> Routines I create a new routine called "BigQuery_libraries". Then I right-click on it -> "Edit Routines Libraries" and with the button "New" I add all my libraries. Finally I went on the job, right-click on it -> "Setup rountines dependencies" and under User routines section I add the routines "BigQuery_libraries". IT WORKS. These libraries are loaded correctly. Thank you again. – SGiux Dec 29 '20 at 11:54

score 0 · Answer 2 · answered Dec 19 '20 at 13:11

0

You should make the appropriate definitions in the pom.xml (use the pom.xml tab in Eclipse to edit the text). See also Maven docs.

  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-bigquery</artifactId>
      <version>1.126.3</version>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-core</artifactId>
      <version>1.94.0</version>
    </dependency>
    <!-- other dependencies -->
  </dependencies>

Maven takes care of adding the dependency to your compile- as well as runtime-Paths.

answered Dec 19 '20 at 13:11

SilverNak

3,283
4
28
44

I add the dependencies in the point 4 using the talend interfaces. I will update the thread – SGiux Dec 19 '20 at 13:13
@SimoneG please consider adding the pom.xml to the question – SilverNak Dec 19 '20 at 13:14
I will do that too. EDIT: I inserted the pom.xml – SGiux Dec 19 '20 at 13:26
SilverNak when you said "Maven takes care of adding the dependency to your compile- as well as runtime-Paths" it's possible that the problem is caused by the impossibility to use Internet on the machine that I'm using? – SGiux Dec 19 '20 at 14:16

score 0 · Answer 3 · answered Dec 28 '20 at 06:41

The dependencies in the pom.xml file are loaded in the maven repository ~/.m2/ (In your case it's ...\TALEND STUDIO\configuration.m2\repository). Mainly used for compilation (with compile scope) & for executing tests (with runtime scope).

While executing the jar, Java is not able to access those jars maybe because those are not in the classpath or in the same directory.

The simplest solution is to place those two new dependency jars you added into the same directory where your application's jar exists or add the path of these jars in the PATH environment variable for java.

Execute the application with the jars visible to java which should resolve the problem.

java.lang.NoClassDefFoundError in Talend Studio

3 Answers3