Broadcast variable Null pointer exception in spark streaming

Question

I have a spark streaming application where I need to access a model saved in a HashMap. I have no problems in running the same code with broadcast variables in the local installation. However I get a null pointer exception when I deploy it on my spark test cluster.

I have stored a model in a HashMap which is serializable. I use a broadcast variables declared as a global static variable to broadcast this hashmap:

public static Broadcast<HashMap<String,FieldModel>> br;
HashMap<String,FieldModel> hm = checkerObj.getModel(esserver, type);
br = ssc.sparkContext().broadcast(hm);

I need to access this model in my mapper phase, and do some operation based on the checkup. The following is a snippet of how I access the broadcast variable.

JavaDStream<Tuple3<Long,Double,String>> split = matched.map(new GenerateType2Scores());

class GenerateType2Scores implements Function<String, Tuple3<Long, Double, String>> {
    @Override
    public Tuple3<Long, Double, String> call(String s) throws Exception{

        Long time = Type2ViolationChecker.getMTS(s);
        HashMap<String,FieldModel> temphm= Type2ViolationChecker.br.value();

        Double score = Type2ViolationChecker.getAnomalyScore(temphm,s);
        return new Tuple3<Long, Double, String>(time,score, s);}
}

The temphm should refer to the hashmap stored in the broadcast variable. Can anyone help me understand what is the correct way to access broadcast variables in JAVA?

I have created a gist to give reference to the code: https://gist.github.com/nipunarora/ed987e45028250248edc

Broadcast variable should not be static. – abalcerek Jun 23 '15 at 16:17 — abalcerek, Jun 23 '15 at 16:17

score 3 · Accepted Answer · answered Jun 23 '15 at 17:25

3

I found the answer thanks to @user52045.

Broadcast variables must be declared final, and cannot be declared static for a global reference :P

answered Jun 23 '15 at 17:25

tsar2512

2,826
3
33
61

me having similar problem , what could be wrong here https://stackoverflow.com/questions/64003697/spark-broadcast-variable-map-giving-null-value – BdEngineer Sep 22 '20 at 08:09
https://stackoverflow.com/questions/63935600/writing-udf-for-looks-up-in-the-map-in-java-giving-unsupported-literal-type-clas – BdEngineer Sep 22 '20 at 08:10
if Broadcast variables must be declared final ,if so will this be accessable inside the UDF without passing as argument ? – BdEngineer Sep 23 '20 at 09:54

Broadcast variable Null pointer exception in spark streaming

1 Answers1