How do I run graphx with Python / pyspark?

Question

I am attempting to run Spark graphx with Python using pyspark. My installation appears correct, as I am able to run the pyspark tutorials and the (Java) GraphX tutorials just fine. Presumably since GraphX is part of Spark, pyspark should be able to interface it, correct?

Here are the tutorials for pyspark: http://spark.apache.org/docs/0.9.0/quick-start.html http://spark.apache.org/docs/0.9.0/python-programming-guide.html

Here are the ones for GraphX: http://spark.apache.org/docs/0.9.0/graphx-programming-guide.html http://ampcamp.berkeley.edu/big-data-mini-course/graph-analytics-with-graphx.html

Can anyone convert the GraphX tutorial to be in Python?

There are some interesting updates on the Apache Jira ticket http://issues.apache.org/jira/browse/SPARK-3789 — Glenn Strycker, Nov 11 '15 at 17:38

score 23 · Answer 1 · edited May 14 '19 at 06:16

You should look at GraphFrames (https://github.com/graphframes/graphframes), which wraps GraphX algorithms under the DataFrames API and it provides Python interface.

Here is a quick example from https://graphframes.github.io/graphframes/docs/_site/quick-start.html, with slight modification so that it works

first start pyspark with the graphframes pkg loaded

pyspark --packages graphframes:graphframes:0.1.0-spark1.6

python code:

from graphframes import *

# Create a Vertex DataFrame with unique ID column "id"
v = sqlContext.createDataFrame([
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
], ["id", "name", "age"])

# Create an Edge DataFrame with "src" and "dst" columns
e = sqlContext.createDataFrame([
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
], ["src", "dst", "relationship"])
# Create a GraphFrame
g = GraphFrame(v, e)

# Query: Get in-degree of each vertex.
g.inDegrees.show()

# Query: Count the number of "follow" connections in the graph.
g.edges.filter("relationship = 'follow'").count()

# Run PageRank algorithm, and show results.
results = g.pageRank(resetProbability=0.01, maxIter=20)
results.vertices.select("id", "pagerank").show()

You could put more explanation other than the links – Ian May 01 '16 at 15:27 — Ian, May 01 '16 at 15:27

score 21 · Accepted Answer · edited Dec 16 '15 at 22:21

21

It looks like the python bindings to GraphX are delayed at least to Spark ~~1.4~~ ~~1.5~~ ∞. It is waiting behind the Java API.

You can track the status at SPARK-3789 GRAPHX Python bindings for GraphX - ASF JIRA

edited Dec 16 '15 at 22:21

zero323

322,348
103
959
935

answered Feb 20 '15 at 21:11

Misty Nodine

326
2
3

Hi, Misty do you have any idea when it will be released ? I have checked it's not available till now even on 1.5.1. – sonus21 Oct 13 '15 at 05:58
1

This is a terrible shame. It seems igraph-python is also partly dead. Is there any other option for handling large graphs in python? – Simd Feb 09 '16 at 18:49

score 3 · Answer 3 · answered Apr 26 '14 at 08:17

3

GraphX 0.9.0 doesn't have python API yet. It's expected in upcoming releases.

answered Apr 26 '14 at 08:17

Wildfire

6,358
2
34
50

So basically GraphX is a Scala-only system since it does not have a Java API either? – Matthew Cornell Nov 04 '14 at 16:26
AFAIK it's still Scala-only – Wildfire Nov 05 '14 at 10:26
2

Actually I think they do have one. See here: https://github.com/amplab/graphx/tree/master/python/examples – A T Jan 29 '15 at 02:03
The original implementation by amplab included a couple of examples, transitive closure and PageRank, but without using the actual GraphX API, just regular PySpark API. GraphX includes a lot of handy functions and classes that are not exposed yet to Python. – Javier de la Rosa Feb 07 '15 at 00:45
I just found that it's done now: https://github.com/kdatta/spark/tree/SPARK-3789/python/pyspark/graphx Maybe it'll be included in the next release. – Javier de la Rosa Feb 07 '15 at 17:15
@JavierdelaRosa That looks like a fork. – Evan Zamir Jul 26 '16 at 21:06
@EvanZamir I think now [GraphFrames](http://graphframes.github.io/) is the way to go. – Javier de la Rosa Jul 27 '16 at 22:14

How do I run graphx with Python / pyspark?

3 Answers3

Linked

Related