3

I have installed the postgresql driver package when I run spark-shell after I have ssh'ed into the EMR spark-shell --packages org.postgresql:postgresql:9.4-1206-jdbc42. I then import org.postgresql. I want to create a spark Dataframe object, so I try to access a table from an s3 instance.

sqlContext.load("jdbc", Map("url" -> "jdbc:postgresql://pathto.table.region.rds.amazonaws.com:5432/table?user=username&password=password","dbtable" -> "table"))

this gives me a java.sql.SQLException: No suitable driver error.

I have looked into this question with a similar problem. But I want to be able to run the driver through spark-shell.

Community
  • 1
  • 1
Jake Fund
  • 395
  • 2
  • 6
  • 16

1 Answers1

2

If I'm not mistaken, (sorry, I don't have a computer to test it now) you'll need to set the driver class explicitly inside your options map. 

sqlContext.load("jdbc", 
         Map("driver" -> "org.postgresql.Driver", 
             "url" -> "jdbc:postgresql://pathto.table.region.rds.amazonaws.com:5432/table?user=username&password=password",
             "dbtable" -> "table"))

PS: I'm writing this answer on my phone, please forgive the eventual typos. But I think that you get the idea.

eliasah
  • 39,588
  • 11
  • 124
  • 154
  • yes thanks, I tried it and it looks like something is wrong with my version of jdbc java.lang.UnsupportedClassVersionError: org/postgresql/Driver : Unsupported major.minor version 52.0 – Jake Fund Apr 09 '16 at 14:23
  • Ok that is another issue. Check the java version between your dependencies, spark and the one installed on emr – eliasah Apr 09 '16 at 14:25