7

This is regarding an issue I am facing while querying Cassandra from Apache Spark.

The normal query from Spark works fine without any issues , however when I query with a condition which is the key I get the below error. Initially I tried querying for a composite key column family and it was also giving the same issue as below.

"Caused by: InvalidRequestException(why:empid cannot be restricted by more than one relation if it includes an Equal)"

Column Family:

CREATE TABLE emp (
empID int,
deptID int,
first_name varchar,
last_name varchar,
PRIMARY KEY (empID));

Column Family Contents:

empID, deptID, first_name, last_name

104, 15, 'jane', 'smith'

Sample SCALA Code:

val job=new Job()
job.setInputFormatClass(classOf[CqlPagingInputFormat])
val host: String = "localhost"
val port: String = "9160"
ConfigHelper.setInputInitialAddress(job.getConfiguration(), host)
ConfigHelper.setInputRpcPort(job.getConfiguration(), port)
ConfigHelper.setInputColumnFamily(job.getConfiguration(), "demodb", "emp")
ConfigHelper.setInputPartitioner(job.getConfiguration(), "Murmur3Partitioner")
CqlConfigHelper.setInputColumns(job.getConfiguration(), "empid,deptid,first_name,last_name")
//CqlConfigHelper.setInputCQLPageRowSize(job.getConfiguration(), limit.toString)
CqlConfigHelper.setInputWhereClauses(job.getConfiguration(),"empid='104'")

// Make a new Hadoop RDD
val casRdd = sc.newAPIHadoopRDD(job.getConfiguration(),
                                classOf[CqlPagingInputFormat],
                                classOf[Map[String, ByteBuffer]],
                                classOf[Map[String, ByteBuffer]])

I kindly request you to let me know if there is any work around for this kind of scenario as I am struggling to overcome this issue for the past few days.

Thanks

Wildfire
  • 6,358
  • 2
  • 34
  • 50
user2081818
  • 309
  • 2
  • 10
  • 1
    Actually I get this using the latest CQL3 devcenter from datastax. In fact I execute the same queries that worked before and now all give this same error. Let me know if you've figure it out! – Adrian Feb 06 '14 at 20:23

1 Answers1

1

This error appears due to the way the query is translated in Cassandra (check org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader#whereClause for details). When the query is translated to Cassandra, it has the following syntax:

SELECT * FROM "emp" WHERE token("empid") > ? AND token("empid") <= ? AND empid='104' LIMIT 1000 ALLOW FILTERING

There is a related JIRA (CASSANDRA-6151) mark as won't fix that discusses some approaches to solve the problem. From the small documentation I have found, CqlConfigHelper.setInputWhereClauses should only be used on indexed columns that are not part of the key.

I hope it helps.

Daniel H.
  • 1,782
  • 16
  • 18