2

I'm currently working on a project using someone else's code. I understand the basic concept of how this code works, but not all of it. To that end, I'm trying to trace a small example through a run. I know I can do this using println but I would really like to visualize the graph after each operation.

Can the graph generated in spark be viewed? I'm using Zeppelin because I read it does help with this and other debugging but I was hoping to actually make a visual representation of the graph.

Is this possible?

Dylan Lawrence
  • 1,503
  • 10
  • 32

1 Answers1

3

Spark GraphX has no built in visualisation method, and Zeppelin provides no built in way to visualise it either. It makes sense that this may have not been a priority if you consider that GraphX (and Spark in general) has as its main purpose to work with huge datasets distributed over a cluster of machines. To visualize the full graph on your machine, you would have to collect the full graph to that single machine, and if you can easily do that, do you really need a distributed system?

If you decide the answer is still yes, there are a few examples of visualising GraphX graphs out there:

Community
  • 1
  • 1
sgvd
  • 3,819
  • 18
  • 31
  • Ok, can I output the graph at each stage right? I don't care what format, I'm thinking, if I can't see it happen with Zeppelin, I could output a graph file of some kind at each stage, convert that to a .dot file, then render all the graphs? – Dylan Lawrence May 24 '16 at 22:13