0

I use socket stream as input stream and then do the map and reduce.

SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(2));
JavaReceiverInputDStream<String> logMessage = jssc.socketTextStream("localhost", 8999);
JavaDStream<Log> logs = logMessage.flatMap(); //function is omitted
JavaPairDStream<String, Integer> pairByTime0 = logs.mapToPair();//function is omitted
JavaPairDStream<String, Integer> r = pairByTime0 .reduceByKey();//function is omitted
ds.print();
jssc.start();              
jssc.awaitTermination();

I've tried to trace the flagMap and make sure that the input is right. I know that every 2s spark will do the job and print the result but why print didn't work.I could not see any results. Here is the log image

Update: Actually the code above works, the problem is in my client program which sends streams to spark.

lionel
  • 415
  • 1
  • 5
  • 14

1 Answers1

-1

Spark operations are lazy. You must call action to get result. It is reason why mapToPair, flatMap and reduceByKey are omitted.

Przemek
  • 208
  • 2
  • 8