0

Pyspark newbie.

Getting error when pyspark workers are executed.

Pyspark script:

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.0.2 pyspark-shell'

from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
import json

sc = SparkContext(appName="PythonSparkStreamingKafka_RM_01")
sc.setLogLevel("WARN")

ssc = StreamingContext(sc, 60)

kafkaStream = KafkaUtils.createStream(ssc, 'localhost:2181', 'test', {'test':1})

parsed = kafkaStream.map(lambda v: v)
parsed.pprint()

ssc.start()
ssc.awaitTermination()

VERSIONS:

Using kafka_2.10-0.8.2.1 and Pyspark version 2.3.0 with Python version 3.6.4 os MacOs sierra

Zookeeper command:

bin/zookeeper-server-start.sh config/zookeeper.properties

Kafka command:

bin/kafka-server-start.sh config/server.properties

Loading some data to queue using

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < /tmp/test_log.log

Verified that test queue indeed receiving data in queue by running below command (data was indeed loaded to test queue)

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

sample test_log.log data

2018-05-01 13:29:54,287 30337 log_generator.py 145 log-generator DEBUG (over headset)
2018-05-01 13:29:54,951 30337 log_generator.py 139 log-generator ERROR metallic surface begin to become visible. A large dish antenna
2018-05-01 13:29:55,876 30337 log_generator.py 143 log-generator WARNING worked with him before. Here he comes.
2018-05-01 13:29:56,196 30337 log_generator.py 139 log-generator ERROR (to Artoo)

Now when I run via spark-submit or python pysparkStreaming.py am getting 

[Stage 0:>                                                          (0 + 1) / 1]2018-05-01 19:43:50 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.AbstractMethodError
:
:
:
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2018-05-01 19:43:50 ERROR TaskSetManager:70 - Task 0 in stage 0.0 failed 1 times; aborting job
2018-05-01 19:43:50 ERROR ReceiverTracker:91 - Receiver has been stopped. Try to restart it.

Complete log https://justpaste.it/522fy

naren
  • 14,611
  • 5
  • 38
  • 45

1 Answers1

0

Probably a version incompatibility issue.

If you're using the latest spark version (2.3) check:

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#creating-streaming-dataframes-and-streaming-datasets

Kafka source - Reads data from Kafka. It’s compatible with Kafka broker versions 0.10.0 or higher. See the Kafka Integration Guide for more details.

raul ferreira
  • 886
  • 7
  • 21