0

I would like to have access to say a JSON file that holds all the details of all the heron instances in a topology (i.e. all the data shown in the heron-ui). I can't find a way to get these details. I am particularly looking for the backpressure metric.

Vinay
  • 952
  • 2
  • 10
  • 27

2 Answers2

1

To access the data shown in the heron-ui, maybe "heron-explorer" is already sufficient?

Link to documentation

Else, heron-tracker has a nice and more or less well documented REST-API.

Link to rest-documentation

However, the examples are very basic and it's quite challenging to find all the metrics available.

Edit: found this Config that tells which metrics are accessible via REST-API

Edit 2: if the tracker does not expose the metric you need, just modify the config file "metrics_sinks.yaml" under the configs for "tmaster-sink". Add the metric you found in the metricsmgr-x.y JSON file (see below) but not in the REST-API.

You could have a look into the logs at .herondata/topologies/[cluster]/[role]/[topology-name]. There you find JSON files named "metrics.json.metricsmgr-x.y". Open them and check if the values of interest are in there (each file holds different data so take the time to explore them a little). Maybe use a JSON parser for readability (JSON Parser). Then you search for your metric name (e.g. __jvm-memory-mb-total) and build your tracker request, e.g (running the tracker locally on my localhost, default environment, local cluster)

   curl "http://localhost:8888/topologies/metrics?cluster=local&environ=default&topology=WordCountStreamletTopology&component=filter1&metricname=__jvm-memory-mb-total"

Don't ignore the underscores "__".

Depending on what you really need the backpressure information for, you might or might not find the relevant metrics, though. And so far, I did not get all metrics listed in the logs via the REST-api, still figuring that out...

Edit: Figured it out: found this Config that tells which metrics are accessible via REST-API

Edit 2: if the tracker does not expose the metric you need, just modify the config file "metrics_sinks.yaml" under the configs for "tmaster-sink". Add the metric you found in the metricsmgr-x.y JSON file (see below) but not in the REST-API.

Make sure to set the configs right (see Configs for instances). E.g., to set the sizes of the queues to avoid or enforce backpressure.

henriette_r
  • 56
  • 3
0

In addition to the answer above, if you just want to find out all the information shown in UI, Heron-UI has endpoints for all of them. You can find these end-points by checking the network requests in your browser.

For examples,

to get the logical plan of the topology: http://HERONUI-SERVICE/topologies/CLUSTER/ENV/TOPOLOGY/logicalplan.json

to get the values of metric __complete-latency for the past 3 hours: http://HERONUI-SERVICE/topologies/metrics?cluster=CLUSTER&environ=ENV&topology=TOPOLOGY&component=COMPONENT&metricname=__complete-latency/default&interval=10800

New end points could be added if needed. Here is the source code for the handlers of all the available end points in Heron UI.

https://github.com/apache/incubator-heron/blob/master/heron/tools/ui/src/python/main.py#L74

Ning Wang
  • 96
  • 3