5

I have a DynamoDB table that I need to connect to EMR Spark SQL to run queries on the table. I got the EMR Spark Cluster with release label emr-4.6.0 and Spark 1.6.1 on it.

I am referring to the document: Analyse DynamoDB Data with Spark

After connecting to the master node, I run the command:

spark-shell --jars /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar

It gives a warning:

Warning: Local jar /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar does not exist, skipping.

Later, when I import the DynamoDB Input Format using

import org.apache.hadoop.dynamodb.read.DynamoDBInputFormat import org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat

It gives the error:

 error: object dynamodb is not a member of package org.apache.hadoop
     import org.apache.hadoop.dynamodb.read.DynamoDBInputFormat
 error: object dynamodb is not a member of package org.apache.hadoop
     import org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat

I think it is the jar that is causing this error. Where do I get this emr-ddb-hadoop.jar?

Shweta
  • 137
  • 1
  • 11

3 Answers3

3

don't use spark-shell --jars, configuration in spark-default.cnf:

spark.driver.extraClassPath  /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar
spark.executor.extraClassPath /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar

Later, import the DynamoDB Input Format is OK

import org.apache.hadoop.dynamodb.read.DynamoDBInputFormat
import org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat
Joe Chien
  • 31
  • 1
  • In case anyone is wondering, typically the location of said configuration file is /etc/spark/conf/spark-defaults.conf – Ivan Gozali Sep 02 '16 at 00:24
2

The root cause of this problem is that emr-ddb-hadoop.jar is not available in the environment (or the location specified). In oder to install the dynamo DB libraries you have to select Hadoop 2.7.2 along with your applications of interest when you are creating the spark EMR cluster. Did you select that ?

If not launch a new cluster, go to advanced options and make sure Hadoop 2.7.2 is selected along with other applications.

user479151
  • 105
  • 2
  • 2
  • 6
0

Indeed /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar will not exist on EMR launch. You should install the necessary bigtop package first:

sudo yum install -y /var/aws/emr/packages/bigtop/emr-ddb/noarch/emr-ddb-hadoop-3.0.0-1.amzn1.noarch.rpm

Notice, probably another package version may be present - check content of the /var/aws/emr/packages/bigtop/emr-ddb/noarch/ directory.

Serge
  • 1,947
  • 3
  • 26
  • 48