4

Can anyone help me with starting spark thrift server? I am running my script in standalone mode and I want to fetch data in my business intelligence tool.

In order to do that I need to start thrift server. I tried running shell script: 

$SPARK_HOME/sbin/start-thriftserver.sh

but I get an error:

error "ps unknown option --0"

mrsrinivas
  • 34,112
  • 13
  • 125
  • 125
Bhanuday Birla
  • 969
  • 1
  • 10
  • 23
  • Provide more debug info, start server with `bash -x sbin/start-thriftserver.sh` and attach log to the question. – Mariusz Nov 17 '16 at 11:14
  • --ps: unknown option -- o (NewLine)Try `ps --help' for more information. starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /e/Development/spark-2.0.1-bin-hadoop2.7/logs/spark--org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-TEAMTAPPS.out After this below line keeps up repeating ps: unknown option -- o Try `ps --help' for more information. – Bhanuday Birla Nov 17 '16 at 11:28
  • 1
    Possible duplicate of http://stackoverflow.com/questions/36593446/failed-to-start-master-for-spark-in-windows – Mariusz Nov 17 '16 at 13:46
  • @mariusz ...the link provides solution to start master but i want to start thrift server. – Bhanuday Birla Nov 18 '16 at 07:32
  • Yes, but this applies to all server scripts, spark-thriftserver is one of them. – Mariusz Nov 18 '16 at 08:35
  • @Mariusz.. I have called the thrift server class directly form cmd as: `E:\Development\spark-2.0.1-bin-hadoop2.7\bin>spark-class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2` but throwing error :`16/11/22 16:29:40 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: A master URL must be set in your configuration` – Bhanuday Birla Nov 22 '16 at 11:03
  • OK, I found a solution that should work in your environment and summarized discussion in the answer. – Mariusz Nov 22 '16 at 12:14

2 Answers2

9

In Spark 2.2.1

cd %SPARK_HOME%\bin
spark-class org.apache.spark.deploy.SparkSubmit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 spark-internal

this started the spark thrift server in windows. In another terminal run the below beeline command to open a jdbc connection

cd %SPARK_HOME%\bin
beeline -u jdbc:hive2://localhost:10000
Remis Haroon - رامز
  • 3,304
  • 4
  • 34
  • 62
  • Hi, I get INFO ThriftCLIService:98 - Starting ThriftBinaryCLIService on port 10000 with 5...500 worker threads. and it hangs there. Do you know why? – user3344443 Dec 01 '18 at 15:05
  • Just to help others... we need t have the right version of winutils in place. – user3103957 Jan 05 '20 at 08:44
5

The sbin/ scripts work fine under linux, but they are not prepared to run in Windows (see Failed to start master for Spark in Windows). But spark thriftserver can be start in foreground independent from OS using command:

java -cp conf/:jars/* org.apache.spark.deploy.SparkSubmit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 spark-internal

Then you can test connection using beeline supplied with spark:

$ ./bin/beeline -u jdbc:hive2://localhost:10000
Connecting to jdbc:hive2://localhost:10000
16/11/22 13:09:57 INFO Utils: Supplied authorities: localhost:10000
16/11/22 13:09:57 INFO Utils: Resolved authority: localhost:10000
16/11/22 13:09:57 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://localhost:10000
Connected to: Spark SQL (version 2.0.1)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://localhost:10000> show databases;
+---------------+--+
| databaseName  |
+---------------+--+
| default       |
| elo           |
+---------------+--+
2 rows selected (0,26 seconds)
Community
  • 1
  • 1
Mariusz
  • 13,481
  • 3
  • 60
  • 64
  • Thank you very much Mariusz. Its a great relief that now i am able to start thrift server. – Bhanuday Birla Nov 22 '16 at 14:06
  • Hi, I get INFO ThriftCLIService:98 - Starting ThriftBinaryCLIService on port 10000 with 5...500 worker threads and it hangs there forever. Do you know why? – user3344443 Dec 01 '18 at 15:06