1

I try to get the min value from javaRDD and delete it.then get the next value but I get the ex min value. for example in sample below the value of pt_min doesn't change.

public class Main {
    static Double min= Double.MAX_VALUE;
    static String pt_min="";

    static JavaRDD<String> input;


    public static void main(String[] a){
        Logger.getLogger("org").setLevel(Level.OFF);
        Logger.getLogger("akka").setLevel(Level.OFF);

        String inputFile = "/home/k/Desktop/exemple/test";

        SparkConf conf = new SparkConf().setAppName("test").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);

        input = sc.textFile(inputFile);

        input.foreach(x-> System.out.println(x));
        pt_min=getMin(input);
        System.out.println("********** "+pt_min);
        input= input.filter(x->! x.equals(pt_min));
        input.foreach(x-> System.out.println(x));
        pt_min=getMin(input);
        System.out.println("********** "+pt_min);
    }

    private static String getMin(JavaRDD<String> input){
        input.foreach(x->{
            if(min>Double.parseDouble(x)) {
                min = Double.parseDouble(x);
                pt_min=x;
            }
        });
    return pt_min;
    }
}

The result is:

12
7
1
2
9
********** 1
12
7
2
9
********** 1

Please help me to fix it!

Esterlinkof
  • 1,444
  • 3
  • 22
  • 27
user9467051
  • 118
  • 2
  • 11

1 Answers1

0

First of all you should reinitialise min field to Double.MAX_VALUE

input.foreach(x-> System.out.println(x));
pt_min=getMin(input);
System.out.println("********** "+pt_min);
input= input.filter(x->! x.equals(pt_min));

// reset min
min= Double.MAX_VALUE;
input.foreach(x-> System.out.println(x));
pt_min=getMin(input);
System.out.println("********** "+pt_min);

And you should use cache in method getMin(input) because input was not saved and you should use cache.

input.cache().foreach(x->{
    if(min>Double.parseDouble(x)) {
        min = Double.parseDouble(x);
        pt_min=x;
    }
});
Esterlinkof
  • 1,444
  • 3
  • 22
  • 27