Calling JDBC to impala/hive from within a spark job and creating a table

Question

I am trying to write a spark job in scala that would open a jdbc connection with Impala and let me create a table and perform other operations.

How do I do this? Any example would be of great help. Thank you!

Hi , did you achieved this ? if so can you please share – sri hari kali charan Tummala Jan 27 '16 at 12:18 — sri hari kali charan Tummala, Jan 27 '16 at 12:18

score 16 · Accepted Answer · answered Aug 18 '15 at 13:10

16

val JDBCDriver = "com.cloudera.impala.jdbc41.Driver"
val ConnectionURL = "jdbc:impala://url.server.net:21050/default;auth=noSasl"

Class.forName(JDBCDriver).newInstance
val con = DriverManager.getConnection(ConnectionURL)
val stmt = con.createStatement()
val rs = stmt.executeQuery(query)

val resultSetList = Iterator.continually((rs.next(), rs)).takeWhile(_._1).map(r => {
    getRowFromResultSet(r._2) // (ResultSet) => (spark.sql.Row)
}).toList

sc.parallelize(resultSetList)

answered Aug 18 '15 at 13:10

Ian

218
3
11

4

awesome thanks for the great example! Do you have any info on how this would look in Python? – alfredox Sep 08 '16 at 20:57
what a lot of folks are looking for – thebluephantom Oct 29 '17 at 21:34

Calling JDBC to impala/hive from within a spark job and creating a table

1 Answers1

Linked