How can I use graphframes with pyspark on AWS EMR?

Question

I'm trying to use the graphframes package in pyspark in Jupyter Notebook (using Sagemaker and sparkmagic) on AWS EMR. I've tried adding a configuration option when creating the EMR cluster in the AWS console:

[{"classification":"spark-defaults", "properties":{"spark.jars.packages":"graphframes:graphframes:0.7.0-spark2.4-s_2.11"}, "configurations":[]}]

But I still got an error when trying to use the graphframes package in my pyspark code in jupyter notebook.

Here's my code (it's from the graphframes example):

# Create a Vertex DataFrame with unique ID column "id"
v = spark.createDataFrame([
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
], ["id", "name", "age"])
# Create an Edge DataFrame with "src" and "dst" columns
e = spark.createDataFrame([
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
], ["src", "dst", "relationship"])
# Create a GraphFrame
from graphframes import *
g = GraphFrame(v, e)

# Query: Get in-degree of each vertex.
g.inDegrees.show()

# Query: Count the number of "follow" connections in the graph.
g.edges.filter("relationship = 'follow'").count()

# Run PageRank algorithm, and show results.
results = g.pageRank(resetProbability=0.01, maxIter=20)
results.vertices.select("id", "pagerank").show()

And here's the output/error:

ImportError: No module named graphframes

I read through this git thread but all the potential work-arounds seem very complicated and require ssh-ing into the master node of the EMR cluster.

Answer 1

I finally figured out that there is a PyPi package for graphframes. I used this to create a bootstrapping action as detailed here, although I changed things a little bit.

Here's what I did to get graphframes working on EMR:

1. First I created a shell script and saved it so s3 named "install_jupyter_libraries_emr.sh":

#!/bin/bash

sudo pip install graphframes

2. I then went through the advanced options EMR creation process in the AWS console.
   • During Step 1, I added in the maven coordinates of the graphframes package within the edit software settings text box:

   [{"classification":"spark-defaults","properties":{"spark.jars.packages":"graphframes:graphframes:0.7.0-spark2.4-s_2.11"}}]
   

   • During Step 3: General Cluster Settings, I went into the bootstrap actions section
   • Within the bootstrap actions section, I added a new custom boostrap action with:
     • an arbitrary name
     • The s3 location of my "install_jupyter_libraries_emr.sh" script
     • no optional arguments
   • I then started the cluster creation
3. Once my cluster was up, I got into Jupyter and ran my code:

# Create a Vertex DataFrame with unique ID column "id"
v = spark.createDataFrame([
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
], ["id", "name", "age"])
# Create an Edge DataFrame with "src" and "dst" columns
e = spark.createDataFrame([
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
], ["src", "dst", "relationship"])
# Create a GraphFrame
from graphframes import *
g = GraphFrame(v, e)

# Query: Get in-degree of each vertex.
g.inDegrees.show()

# Query: Count the number of "follow" connections in the graph.
g.edges.filter("relationship = 'follow'").count()

# Run PageRank algorithm, and show results.
results = g.pageRank(resetProbability=0.01, maxIter=20)
results.vertices.select("id", "pagerank").show()

And this time, finally, I got the correct output:

+---+--------+
| id|inDegree|
+---+--------+
|  c|       1|
|  b|       2|
+---+--------+

+---+------------------+
| id|          pagerank|
+---+------------------+
|  b|1.0905890109440908|
|  a|              0.01|
|  c|1.8994109890559092|
+---+------------------+

Answer 2

@Bob Swain's answer is fine, but now the repositories for graphframes are located at https://repos.spark-packages.org/. Thus, in order to make it work the classification should be changed to:

[
 {
  "classification":"spark-defaults",
  "properties":{
    "spark.jars.packages":"graphframes:graphframes:0.8.0-spark2.4-s_2.11",
    "spark.jars.repositories":"https://repos.spark-packages.org/"
  }
 }
]

Answer 3

My requirement: To Map the property graph model in Hadoop (AWS EMR) and store it somewhere so we can query later.

Is there any option to get the result in as a graph also? I have read few documents on Graphx which is a library supported by APache spark , I don't have clear idea