Object sql is not a member of package org.apache.spark

Question

I am trying to work with spark-sql but while I importing

import org.apache.spark.sql.{Row, SparkSession}

getting following error:

object sql is not a member of package org.apache.spark

Here are my details:

Spark version: 1.6.2 Scala version: 2.11.8 sbt version: 0.13.16

Here is my build.sbt file:

name := "sbt_demo"
version := "1.0"
scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.6.2"

Note: there is another question with the same problem in Stack Overflow, but that accepted answer didn't help me. That is the reason why I am asking again.

@RameshMaharjan why are asking like that? is spark 1.6.2 is not a stable version? — user6325753, Dec 05 '17 at 16:11
you should use 2.2.0 for the latest and your problem should be solved I guess — Ramesh Maharjan, Dec 05 '17 at 16:13
Are you using some sort of IDE? Perhaps dependencies are not reloaded automatically. Here you find a helpful answer if you are using IntelliJ: https://stackoverflow.com/questions/20413605/how-to-force-intellij-idea-to-reload-dependencies-from-build-sbt-after-they-chan — stefanobaghino, Dec 05 '17 at 16:13
@stefanobaghino, i am using eclipse scala ide, and i build the project in ide and as well as in terminal by using sbt reload.. still no use.. — user6325753, Dec 05 '17 at 16:15
@RameshMaharjan I'm not sure that's the problem, `org.apache.spark.sql` was part of the distribution with `1.6.2` already (but using a more recent version is still a reasonable advice, if possible). https://spark.apache.org/docs/1.6.2/api/scala/index.html#org.apache.spark.sql.package — stefanobaghino, Dec 05 '17 at 16:16
@RameshMaharjan , but i want to know the reason why i am not able to import the sql statement even after dependency added. — user6325753, Dec 05 '17 at 16:17
Two minor things: in SBT 0.x it's advisable to put blank lines in between each line (it used to be necessary). Also, can you try to remove the dependency to `spark-core`? It's brought in transitively by `spark-sql` and I'm not fully sure if adding two separate libraries like that correctly imports both JARs (usually the `++=` operator is used with a `Seq` of libraries, instead of adding them one by one). — stefanobaghino, Dec 05 '17 at 16:19
your import statement suggests that you want to use SparkSession which was introduced in 2.0 and I guess there was no sql package for sql in 1.6.2 — Ramesh Maharjan, Dec 05 '17 at 16:20
@RameshMaharjan, i have tried to import SQLContext which is available in spark 1.6.2, still same issue. — user6325753, Dec 05 '17 at 16:22
@stefanobaghino, i tried in that way also getting same issue. — user6325753, Dec 05 '17 at 16:23
I've just been able to copy and paste your SBT definition and open a `console` with `sbt` 0.13.16 and I could import `org.apache.spark.sql._` without any problem. There is probably something else wrong. Can you run `sbt clean` and check that all JARs are downloaded as expected? — stefanobaghino, Dec 05 '17 at 16:25

score 2 · Answer 1 · answered Jun 19 '18 at 07:20

I had the same problem and this is my build.sbt file

name := "rec"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"

some says this command below will work

sbt reload package    
sbt update 
sbt reload

but it didn't work for me. so i deleted .idea file and re-import everything in build.sbt file and it works well for me

score 1 · Answer 2 · answered Dec 07 '17 at 06:08

For sbt you can use

"org.apache.spark" %% "spark-sql" % "1.6.2" % "provided",

For Maven

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.10</artifactId>
        <version>1.6.2</version>
    </dependency>

Use below import just before when u create dataframe . inside your code

import sqlContext.implicits._
val df = sqlContext.createDataFrame(rows, schema)

score 0 · Answer 3 · answered Dec 05 '17 at 16:24

0

You need maven central in your Resolver of sbt. And if behind proxy, set your proxy properly for SBT.

Also, in spark 1.6.2, there's no SparkSession... You should use SQLContext, or move to 2.x

answered Dec 05 '17 at 16:24

Adrian

21
5

Can you please elaborate your answer? " You need maven central in your Resolver of sbt. And if behind proxy, set your proxy properly for SBT" i didn't understand. – user6325753 Dec 05 '17 at 16:26
@user6325753 can you paste your ivy-setting.xml, along with your network conditions? – Adrian Dec 06 '17 at 02:00

Object sql is not a member of package org.apache.spark

3 Answers3