Stack Overflow
Stack Overflow

Questions tagged [spark-cloudant]

Cloudant integration with Spark as Spark SQL external datasource, and Spark Streaming as a custom receiver.

Cloudant integration with Spark as Spark SQL external datasource, and Spark Streaming as a custom receiver. For more info, see: https://github.com/cloudant-labs/spark-cloudant

7 questions
3
votes
2 answers

Why does sbt assembly in Spark project fail with "Please add any Spark dependencies by supplying the sparkVersion and sparkComponents"?

I work on a sbt-managed Spark project with spark-cloudant dependency. The code is available on GitHub (on spark-cloudant-compile-issue branch). I've added the following line to build.sbt: "cloudant-labs" % "spark-cloudant" % "1.6.4-s_2.10" %…
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
1
vote
1 answer

calculating skewness using spark.sql and Cloudant

I have problems with the following piece of code: def skewTemperature(cloudantdata,spark): return spark.sql("""SELECT (1/count(temperature)) * (sum(POW(temperature-%s,3))/pow(%s,3)) as skew from washing"""…
Navin Deshpande
  • 21
  • 2
1
vote
2 answers

Converting sensor tag data in DSX

I'm working on converting the existing recipe for Data Science Experience (DSX) to use data from a connected Sensor Tag device. However the mobile applications for that device send the data as strings rather than numerics - this is causing the DSX…
Skilganon
  • 33
  • 4
1
vote
1 answer

Spark Cloudant error: 'nothing was saved because the number of records was 0!'

I'm using the spark-cloudant library 1.6.3 that is installed by default with the spark service. I'm trying to save some data to Cloudant: val df = getTopXRecommendationsForAllUsers().toDF.filter( $"_1" > 6035) println(s"Saving ${df.count()} ratings…
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
1
vote
2 answers

Spark-cloudant package 1.6.4 loaded by %AddJar does not get used by notebook

I'm trying to use the latest spark-cloudant package with a notebook: %AddJar -f https://github.com/cloudant-labs/spark-cloudant/releases/download/v1.6.4/cloudant-spark-v1.6.4-167.jar Which outputs: Starting download from…
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
0
votes
1 answer

What is the solution for the error, “JBlas is not a member of package or apache”?

I tried to solve it from both of these ( this and this) threads, and it worked for me on my own virtual machine but didn’t work in cloud dataproc. I did the same process for both of them. But there is still error in the cloud which is same as the…
santobedi
  • 866
  • 3
  • 17
  • 39
0
votes
1 answer

com.cloudant.spark data source not found in DSX Notebook

I'm trying to follow https://developer.ibm.com/clouddataservices/docs/ibm-data-science-experience/docs/load-and-filter-cloudant-data-with-spark/ to load cloudant data with Spark. I have a Scala 2.11 with Spark 2.1 (happens with Spark 2.0 as well)…
Wieland
  • 1,663
  • 14
  • 23