2

I am trying to have a PCollection of AutoValue-defined objects that I have created, and I've added the appropriate annotations to infer the Schema via DefaultSchema(AutoValueSchema.class). Like so:

@DefaultSchema(AutoValueSchema.class)
@AutoValue
public abstract class MyAutoClas {
  public abstract String getMyStr();
  public abstract Integer getMyInt();

  @CreateSchema
  public static MyAutoClass create(String myStr, Integer myInt) {
    return new AutoValue_MyAutoClass(myStr, myInt);
  }
}

I have a small test case that looks like this:

PCollection<KV<String, MyAutoClass>> result = pipeline
    .apply(Create.of(MyAutoClass.create("abc", 1)))
    .apply(WithKeys.of(in -> in.getMyStr()));

PAssert.that(result).containsInAnyOrder(KV.of("abc", MyAutoClass.create("abc", 1)));
pipeline.run().waitUntilFinish();

When I try to run this, I am seeing the following errors:

[ERROR] testMyAutoValueClass(.....)  Time elapsed: 1.891 s  <<< ERROR!
java.lang.RuntimeException: Creator parameter arg0 Doesn't correspond to a schema field
    at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$InvokeUserCreateInstruction.<init>(ByteBuddyUtils.java:717)
    at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$StaticFactoryMethodInstruction.<init>(ByteBuddyUtils.java:660)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.createStaticCreator(JavaBeanUtils.java:284)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.lambda$getStaticCreator$4(JavaBeanUtils.java:273)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1664)
    at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.getStaticCreator(JavaBeanUtils.java:269)
    at org.apache.beam.sdk.schemas.AutoValueSchema.lambda$schemaTypeCreatorFactory$673bce5b$1(AutoValueSchema.java:80)
    at org.apache.beam.sdk.schemas.UserTypeCreatorFactory.create(UserTypeCreatorFactory.java:21)
  ....... [ETCETERA] ......
Pablo
  • 10,425
  • 1
  • 44
  • 67

1 Answers1

7

This error occurs because ByteBuddy and the Java reflection utilities are not able to infer the parameter names of your @SchemaCreate method (thus complaining about some unknown parameter arg0 - this is the default name).

To ensure Java reflection can find the parameter names, you need to instruct the compiler to include this information. If you are building with Maven, you can do it this way:

<properties>
    <!-- PLUGIN VERSIONS -->
    <maven-compiler-plugin.version>3.1</maven-compiler-plugin.version>

    <!-- OTHER PROPERTIES -->
    <java.version>1.8</java.version>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>${maven-compiler-plugin.version}</version>
            <configuration>
                <!-- Original answer -->
                <compilerArgument>-parameters</compilerArgument>
                <!-- Or, if you use the plugin version >= 3.6.2 -->
                <parameters>true</parameters>
                <testCompilerArgument>-parameters</testCompilerArgument>
                <source>${java.version}</source>
                <target>${java.version}</target>
            </configuration>
        </plugin>
    </plugins>
</build>

Note: To accomplish the same thing with Gradle, use this answer.

Pablo
  • 10,425
  • 1
  • 44
  • 67
  • 1
    For how to set this in Intellij: https://stackoverflow.com/questions/39217830/how-to-use-parameters-javac-option-in-intellij For gradle users: https://stackoverflow.com/questions/37463902/how-to-pass-parameters-javac-flag-to-java-compiler-via-gradle – Valentyn May 29 '20 at 00:41
  • I didn't require further settings in IntelliJ. Simply adding the above to Maven worked a treat. Do make sure you comment out the config that you don't need for your plugin version. – DataMacGyver Dec 03 '20 at 09:04