0

I am using https://github.com/DmitryBe/clickhouse-spark-connector

I create my jar with sbt assembly after I cloned the repo and then I add my import statements.

import io.clickhouse.ext.ClickhouseConnectionFactory
import io.clickhouse.ext.spark.ClickhouseSparkExt._

object clickhouse is not a member of package spark.jobserver.io

I can see that these paths exist and they are added as dependencies the same way I have added all the others. I have cleaned and rebuilt etc but it has made no difference. I am using scala-ide(eclipse).

ozzieisaacs
  • 833
  • 2
  • 11
  • 23
  • Try to invalidate cache and restart IDE. That should work I guess – Ramesh Maharjan May 26 '17 at 00:06
  • I cant find how to clear the cache in scala ide - I have run clean and restarted it several times and it hasn't helped. – ozzieisaacs May 26 '17 at 15:04
  • try this http://eclipsesource.com/blogs/2012/08/31/reloading-your-p2-cache/ – Ramesh Maharjan May 26 '17 at 15:21
  • I'm pretty sure the module is just broken. I imported all the classes manually and I can't get one of the dependencies to resolve no matter what I do. Thank you for your help. I also loaded up intellij and tried it there to make sure it wasn't a caching issue. – ozzieisaacs May 26 '17 at 17:08
  • If you update your question with your requirements, sample codes, then I including others would be helping you quickly – Ramesh Maharjan May 26 '17 at 17:14

1 Answers1

0

You can try using https://github.com/yandex/clickhouse-jdbc

Here is a snippet which you can use to write dataframe into Clickhouse using your own dialect. ClickhouseDialect is a class which extends JdbcDialects. You can create your dialect and register it using JdbcDialects.registerDialect(clickhouse)

def write(data: DataFrame, jdbcUrl: String, tableName: String): Unit = {

    val clickhouse = new ClickhouseDialect()

    JdbcDialects.registerDialect(clickhouse)

    val props = new java.util.Properties()
    props.put("driver", "ru.yandex.clickhouse.ClickHouseDriver")
    props.put("connection", jdbcUrl)

    val repartionedData = data.repartition(100)

    repartionedData.write
      .mode(SaveMode.Append)
      .jdbc(jdbcUrl, tableName, props)

    JdbcDialects.unregisterDialect(clickhouse)

}

You can check here to create your own dialect. You might have to override canHandle, getJDBCType, getCatalystType methods of JdbcDialects for your usage.

Jigar Mehta
  • 73
  • 3
  • 10