How to override method in uber jar?

Question

I am trying to use this code with Google Cloud Datastore:

Query<Entity> query = Query.entityQueryBuilder()
    .kind("Task")
    .filter(PropertyFilter.hasAncestor(
        datastore.newKeyFactory().kind("TaskList").newKey("default")))
    .build();
datastore.run(query, ReadOption.eventualConsistency());

I get this error:

Exception in thread "main" java.lang.VerifyError: class com.google.datastore.v1.ReadOptions$Builder overrides final method mergeUnknownFields.(Lcom/google/protobuf/UnknownFieldSet;)Lcom/google/protobuf/GeneratedMessage$Builder; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at com.google.datastore.v1.ReadOptions.toBuilder(ReadOptions.java:392) at com.google.datastore.v1.ReadOptions.newBuilder(ReadOptions.java:386) at com.google.cloud.datastore.DatastoreImpl.toReadOptionsPb(DatastoreImpl.java:217) at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:78)

Some general details:

Language: scala
Run on: Google compute engine
Using com.google.guava:guava:19.0 and com.google.cloud:google-cloud:0.3.0 dependencies
This is the code from google.datastore.v1.protos

public final Builder mergeUnknownFields( final com.google.protobuf.UnknownFieldSet unknownFields) { return this; }

this is the code from com.google.protobuf.GeneratedMessage

    public BuilderType mergeUnknownFields(UnknownFieldSet unknownFields) {
    this.unknownFields = UnknownFieldSet.newBuilder(this.unknownFields).mergeFrom(unknownFields).build();
    this.onChanged();
    return this;
}

`

score 1 · Answer 1 · answered Sep 19 '16 at 08:56

Adding Shaded Dependencies solved the spark/google cloud client api conflict problem by using:

<plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.3</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>com.google</pattern>
                                <shadedPattern>shaded.com.google</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>

score 0 · Answer 2 · answered Sep 15 '16 at 16:36

0

I believe this is a symptom of the issues described in this issue.

answered Sep 15 '16 at 16:36

Ed Davisson

2,927
11
11

Yes I also believe this issue is causal by versions mismatch of the ptotobuf protocol in my case between spark and Google cloud Client api (the same proc run on the same cluster with without spark. Unfortunatly, using Google cloud dataproc service (for parallel processsing) the spark classes are preloaded from the service image. – Ron Topol Sep 16 '16 at 08:48

How to override method in uber jar?

2 Answers2