2

I have a POJO defined in Java

@Data
@Builder
public class JavaItem {
    private String name;

}

I have the code in Scala

case class Record(name: String)

def asJavaItem(record: Record): JavaItem = {
    JavaItem.builder().build()
}

def recordDatasetToListJavaItem(record: Dataset[Record]): java.util.List[JavaItem] = {
    implicit val encoder: Encoder[JavaItem] = Encoders.bean(classOf[JavaItem])
    record.map(asJavaItem).collectAsList() // this fails
}

val recordDataset = Seq(Record("name")).toDS()

recordDatasetToListJavaItem(recordDataset)

I'm getting this error message:

org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 24, Column 11: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 24, Column 11: No applicable constructor/method found for zero actual parameters; candidates are: "JavaItem(java.lang.String)"

Why am I getting this error? I suspect this is a problem with encoders. How do I correctly map the Dataset[Record] to a list of JavaItem ?

Gaël J
  • 11,274
  • 4
  • 17
  • 32
AznBoyStride
  • 305
  • 2
  • 12
  • I doubt Lombok and Spark will work nicely together without effort. Is there any reason not to collect as a list of `Record` and map it to `JavaItem` afterwards? – Gaël J Jul 01 '23 at 19:39
  • @GaëlJ I've been using Spark and Lombok for quite some time and didn't have any issues (as long as all required annotations are present) – werner Jul 01 '23 at 19:51
  • Good to know @werner – Gaël J Jul 02 '23 at 07:14

1 Answers1

2

You need a no-args constructor on the Java pojo. Spark occasionally constructs the objects first and then later calls the setters for the single data fields. That's why you need a constructur public JavaItem(){}.

The following Lombok annotations work:

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class JavaItem {
    private String name;
}

The all-args constructor is required as - according to this answer- the builder only adds an all-args constructur if no other constructor is present..

werner
  • 13,518
  • 6
  • 30
  • 45
  • Thanks. You are correct. In my actual Java POJO that I'm using, there is a nested structure. Every POJO in the structure needs the NoArgsConstructor and AllArgsConstructor. (Just an FYI for others that reads this solution) – AznBoyStride Jul 01 '23 at 20:54