How to make this SpringML (or other) Spark SFTP Server connector work?

Question

I am trying to build a Spark Job which can connect to a SFTP Server and drop a csv file there. The option that I found was to use the SpringML package. However, I keep getting a noSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;. This was once described in an issue related to the package.

It is not exactly clear to me how the issue was resolved and if I am just making a user error. I use the newest version of the SpringML package (1.0.3) on Databricks and installed it through the Databricks Maven in "Create Library".

The code I use looks as follows:

// Read sample dataframe from table
val df = sqlContext.sql("SELECT * FROM default.some_test_data")

// Write sample data to a SFTP server
df.write.format("com.springml.spark.sftp").
           option("host", "SFTP_HOST").
           option("username", "SFTP_USER").
           option("password", "password").
           option("fileType", "fileType")
          .save(/some_test_data.csv)

I am happy with any working example, even if it is using another open source package. Also feel free to point out what I miss?

Note: My Spark Version is 3.1.2, Scala 2.12

https://stackoverflow.com/questions/75947449/run-a-scala-code-jar-appear-nosuchmethoderrorscala-predef-refarrayops — Dmytro Mitin, Apr 07 '23 at 05:00

score 0 · Accepted Answer · answered Apr 13 '22 at 13:16

0

There is no support for Spark 3.0 & Scala 2.12 yet.

answered Apr 13 '22 at 13:16

MANI PRAKASH

46
3

1

Yeah in the end I realized and we went with an implementation of SFTP connection via Redsocks. – DataBach Apr 13 '22 at 14:56
Would you mind sharing steps you followed to integrate Spark & Redsocks? – MANI PRAKASH Apr 19 '22 at 09:57

How to make this SpringML (or other) Spark SFTP Server connector work?

1 Answers1