How to check the Spark version

Question

as titled, how do I know which version of spark has been installed in the CentOS?

The current system has installed cdh5.1.0.

score 106 · Accepted Answer · answered Apr 17 '15 at 05:24

106

If you use Spark-Shell, it appears in the banner at the start.

Programatically, SparkContext.version can be used.

answered Apr 17 '15 at 05:24

Shyamendra Solanki

8,751
2
31
25

4

Hmm.. I'm getting `` in python shell – Piko Monde Mar 04 '20 at 03:19
2

@PikoMonde version is a property on the SparkContext class so you just need to call it on an **instance** of that class. – Joshua Ostrom Apr 02 '20 at 10:08
2

Yep, I just realize that. I think, for someone like me, who is new on python and spark, a complete code (programatically) is helpful. Here I wrote the complete code [below](https://stackoverflow.com/a/60523639/9183891) – Piko Monde Apr 02 '20 at 21:01

score 41 · Answer 2 · answered Aug 27 '15 at 17:19

41

Open Spark shell Terminal, run sc.version

answered Aug 27 '15 at 17:19

Venu A Positive

2,992
2
28
31

score 38 · Answer 3 · answered Mar 22 '16 at 16:16

38

You can use spark-submit command: spark-submit --version

answered Mar 22 '16 at 16:16

Ozgur Ozturk

1,265
11
9

mrsrinivas · Answer 4 · 2021-01-07T05:50:06.450

In Spark 2.x program/shell,

use the

spark.version

Where spark variable is of SparkSession object

Using the console logs at the start of `spark-shell`

[root@bdhost001 ~]$ spark-shell
Setting the default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/

Without entering into code/shell

`spark-shell --version`

[root@bdhost001 ~]$ spark-shell --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/
                        
Type --help for more information.

`spark-submit --version`

[root@bdhost001 ~]$ spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/
                        
Type --help for more information.

sometime it will be `spark2-shell --version` or `spark2-submit --version` — mrsrinivas, Sep 17 '19 at 03:24

score 14 · Answer 5 · answered Jan 06 '17 at 00:34

14

If you are using Databricks and talking to a notebook, just run :

spark.version

answered Jan 06 '17 at 00:34

Pat

697
9
12

score 5 · Answer 6 · answered Apr 11 '18 at 08:23

If you are using pyspark, the spark version being used can be seen beside the bold Spark logo as shown below:

manoj@hadoop-host:~$ pyspark
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Python version 2.7.6 (default, Jun 22 2015 17:58:13)
SparkContext available as sc, HiveContext available as sqlContext.
>>>

If you want to get the spark version explicitly, you can use version method of SparkContext as shown below:

>>>
>>> sc.version
u'1.6.0'
>>>

score 4 · Answer 7 · answered Jan 18 '16 at 12:57

Which ever shell command you use either spark-shell or pyspark, it will land on a Spark Logo with a version name beside it.

$ pyspark
$ Python 2.6.6 (r266:84292, May 22 2015, 08:34:51) [GCC 4.4.7 20120313 (Red Hat 4.4.7-15)] on linux2 ............ ........... Welcome to
version 1.3.0

score 4 · Answer 8 · answered Aug 02 '18 at 09:51

4

If you are on Zeppelin notebook you can run:

sc.version

to know the scala version as well you can ran:

util.Properties.versionString

answered Aug 02 '18 at 09:51

HISI

4,557
4
35
51

score 4 · Answer 9 · answered Aug 02 '18 at 09:55

4

use below to get the spark version

spark-submit --version

answered Aug 02 '18 at 09:55

Swift user

69
1
3

score 4 · Answer 10 · answered Jul 09 '20 at 00:12

4

If you want to print the version programmatically use

 from pyspark.sql import SparkSession 

 spark = SparkSession.builder.master("local").getOrCreate() 
 print(spark.sparkContext.version)

answered Jul 09 '20 at 00:12

Julian2611

422
3
11

score 2 · Answer 11 · answered Mar 04 '20 at 10:11

If you want to run it programatically using python script

You can use this script.py:

from pyspark.context import SparkContext
from pyspark import SQLContext, SparkConf

sc_conf = SparkConf()
sc = SparkContext(conf=sc_conf)
print(sc.version)

run it with python script.py or python3 script.py

This above script is also works on python shell.

Using print(sc.version) directly on the python script won't work. If you run it directly, you will get this error:NameError: name 'sc' is not defined.

This should be `sc = SparkContext.getOrCreate(conf=sc_conf)`. _Not_ like this: `sc = SparkContext(conf=sc_conf)`! — toom, Aug 05 '21 at 11:22

score 2 · Answer 12 · edited Oct 11 '21 at 15:02

2

Try this way:

import util.Properties.versionString
import org.apache.spark.sql.SparkSession

val spark = SparkSession
  .builder
  .appName("my_app")
  .master("local[6]")
  .getOrCreate()
println("Spark Version: " + spark.version)
println("Scala Version: " + versionString)

edited Oct 11 '21 at 15:02

Maëlan

3,586
1
15
35

answered Oct 10 '21 at 16:34

Julio Delgado

21
2

score 1 · Answer 13 · edited Jul 08 '20 at 15:58

1

Most of the answers here requires initializing a sparksession. This answer provide a way to statically infer the version from library.

ammonites@ org.apache.spark.SPARK_VERSION
res4: String = "2.4.5"

edited Jul 08 '20 at 15:58

surj

4,706
2
25
34

answered Mar 11 '20 at 05:40

Dyno Fu

8,753
4
39
64

While this code may answer the question, providing additional context regarding why and/or how this code answers the question improves its long-term value. – Igor F. Mar 11 '20 at 09:19

score 1 · Answer 14 · answered May 30 '21 at 16:17

If like me, one is running spark inside a docker container and has little means for the spark-shell, one can run jupyter notebook, build SparkContext object called sc in the jupyter notebook, and call the version as shown in the codes below:

docker run -p 8888:8888 jupyter/pyspark-notebook ##in the shell where docker is installed

import pyspark
sc = pyspark.SparkContext('local[*]')
sc.version

Khetanshu · Answer 15 · 2020-02-13T00:01:57.077

-1

In order to print the Spark's version on the shell, following solution work.

SPARK_VERSION=$(spark-shell --version &> tmp.data ; grep version tmp.data | head -1 | awk '{print $NF}';rm tmp.data)
echo $SPARK_VERSION

edited Feb 13 '20 at 00:01

answered Feb 12 '20 at 23:53

Khetanshu

9
2

Valeriy Solovyov · Answer 16 · 2020-11-24T19:47:13.387

-1

Non-interactive way, that I am using for AWS EMR proper PySpark version installation:

# pip3 install pyspark==$(spark-submit --version 2>&1| grep -m 1  -Eo "([0-9]{1,}\.)+[0-9]{1,}") 
Collecting pyspark==2.4.4

solution:

#  spark-shell --version 2>&1| grep -m 1  -Eo "([0-9]{1,}\.)+[0-9]{1,}"
2.4.4

solution:

# spark-submit --version 2>&1| grep -m 1  -Eo "([0-9]{1,}\.)+[0-9]{1,}"
2.4.4

edited Nov 24 '20 at 19:47

answered Jul 08 '20 at 19:18

Valeriy Solovyov

5,384
3
27
45

It’s using grep and pipe, while non other answer is using non-interactive approach without cache the output in file.There is example how to use it with pip install – Valeriy Solovyov Jul 19 '20 at 05:02

How to check the Spark version

16 Answers16

In Spark 2.x program/shell,

Using the console logs at the start of `spark-shell`

Without entering into code/shell

`spark-shell --version`

`spark-submit --version`

Linked

How to check the Spark version

16 Answers16

In Spark 2.x program/shell,

Using the console logs at the start of spark-shell

Without entering into code/shell

spark-shell --version

spark-submit --version

Linked

Using the console logs at the start of `spark-shell`

`spark-shell --version`

`spark-submit --version`