Stack Overflow
Stack Overflow

Questions tagged [pregel]

8 questions
3
votes
0 answers

"Application attempt...doesn't exist in ApplicationMasterService cache” cause? (Pregel: maxIterations impact on cluster for non-convergent algorithm)

I've tried to run my own Pregel method for a relatively small graph (250k vertices, 1.5M edges). The algorithm which I use may (high chances are) be non-convergent meaning in most cases maxIterations setting is actually acting as hard stop finishing…
pnet_fabric
  • 79
  • 8
3
votes
0 answers

Why my code takes so much time to execute in Spark Pregel?

I have written code with Pregel in Spark which processes a graph, but it executes very very slowly for a small dataset. I have written programs with pregel before, but this code really works slow. my cluster consists of 2 workers. each have core i5…
user13056355
3
votes
1 answer

How to implement cycle detection with pyspark graphframe pregel API

I am trying to implement the algorithm from Rocha & Thatte (http://cdsid.org.br/sbpo2015/wp-content/uploads/2015/08/142825.pdf) with Pyspark and the pregel wraper from graphframes. Here I am getting stuck with the correct syntax for the message…
Alex Ortner
  • 1,097
  • 8
  • 24
1
vote
1 answer

ImportError: cannot import name 'Pregel' from 'graphframes.lib'

I am using pyspark and graphframes from jupyter. I am able to successfully import pyspark and graphframes, but when I try: from graphframes.lib import Pregel I get the following error: ImportError: cannot import name 'Pregel' from…
ChrisDanger
  • 1,071
  • 11
  • 10
1
vote
1 answer

Confused about stop condition on Spark/Graphx/Pregel example program to find 'path distance

' I am working my way through Graphx In Action and this book (source code for which is here: https://github.com/insidedctm/spark-graphx-in-action) discusses two ways of calculating the distance (number of edge hops) between the root of a tree and…
Chris Bedford
  • 2,560
  • 3
  • 28
  • 60
0
votes
0 answers

How to access the processed result content in ArangoDB's Pregel when executing the LinearRank algorithm?

The field in the resulting nodes seem to be stored in a format that I can't query using the web console, Python, or even Java. The content appears to be represented as a string (non-representable type double). I'm unsure if it's stored in VelocyPack…
tuxdroid
  • 13
  • 1
  • 4
0
votes
0 answers

Graph not updating past first iteration of Pregel implementation of an algorithm using Spark GraphX

I am writing a graph algorithm to find path from certain source vertices to target vertices based on a path definition in a graph using Spark's GraphX library's Pregel API. I am using scala version 2.12 with spark standalone cluster version…
gimli_iitk
  • 13
  • 5
0
votes
0 answers

Why is Spark Jobs backing up and fill up memory?

I'm designing a shortest-path algorithm in Spark, using Pregel. I want to partition my graph and try out which partitioning strategy that works best. To do this, I have built a for-loop that is running a strategy five times, recording run-time then…
Gaspegre
  • 11
  • 2