How to filter the null data from Cassandra in Spark?

Question

I want to filter the Null value which selected from Cassandra.

Here is my query:

scala> var rdd =  sc.cassandraTable("keyspace", "table").select("month", "timetag", "name").where("month = ?", "201704")
scala> var data = rdd.filter(_.getString("name") != null)

If I use getString("xxx") compare to null, it will show

"java.lang.NullPointerException: Unexpected null value of column 2. Use get[Option[...]] to receive null values."

After that, I tried to use getStringOption to compare.

scala> var rdd =  sc.cassandraTable("keyspace", "table").select("month", "timetag", "name").where("month = ?", "201704")
scala> var data = rdd.filter(_.getStringOption("name") != null)

This time, it didn't show any error message. But the data is not filtered. The null data is still there.

Does anyone know what is different between getString and getStringOption? Or is there any other way I can compare the data from Cassandra is null?

Thanks a lot!

Assaf Mendelson · Answer 1 · 2017-07-20T12:27:35.030

2

Try doing:

_.getStringOption("name").isDefined

instead of:

 _.getStringOption("name") != null

Update

It might be relevant to expand a little on the Option part.

The getStringOption (which is internally implemented as get[Option[String]] returns a scala Option class.

Option is an idiomatic way in functional programming to represent something which either has some value (denoted as Some(value)) or no value (denoted as None).

Unlike the use of null in java, None is actually an object with various functionality (the example here is isDefined). This makes it easy to combine multiple operation on object while making sure that if something on the way is no defined then we would not get a null pointer exception.

For more information on the Option type see the scala class documentation or this stack overflow answer or this stack overflow answer

edited Jul 20 '17 at 12:27

answered Jul 20 '17 at 07:49

Assaf Mendelson

12,701
5
47
56

It works now!! But how could I know what is the method I can use for getStringOption? Is there any document for something like this? Really thanks!! – Emma Lin Jul 20 '17 at 09:51
@EmmaLin getStringOption returns a scala option. See https://www.scala-lang.org/api/current/scala/Option.html – Assaf Mendelson Jul 20 '17 at 09:51
@AssafMendelson Add please simple explanation that it's an Option and how we should check Options :) – T. Gawęda Jul 20 '17 at 12:14
@T.Gawęda Done. – Assaf Mendelson Jul 20 '17 at 12:27

score 0 · Answer 2 · answered Jul 20 '17 at 08:07

0

Use getOption variants when you expect to receive Cassandra null values

_.get[Option[String]]("name")
res8: Option[String] = Some(abc)

answered Jul 20 '17 at 08:07

Nayan Sharma

1,823
18
19

That is the implementation for getStringOption which the OP used. – Assaf Mendelson Jul 20 '17 at 09:53

How to filter the null data from Cassandra in Spark?

2 Answers2