How to query the kubelet metrics API in Java?

Question

What's the proper way to query the prometheus kubelet metrics API using Java, specifically the PVC usage metrics? I can see there's an API to fetch some metrics via the auto-scaler, but my cluster doesn't have an auto-scaler, so this returns an empty list:

AutoscalingV2beta2Api autoscalingV2beta2Api = new AutoscalingV2beta2Api(apiClient);
var autoscalers = autoscalingV2beta2Api
    .listHorizontalPodAutoscalerForAllNamespaces(null, null, null, null, null, null, null, null, null)
    .getItems();

For reference, I think this is what I need to fetch: https://stackoverflow.com/a/56870054/18437 I'm not keen on using the kubectl -n <namespace> exec <pod-name> df method because then I'll need to make the pod volume mounts back to the volume claims list I've already fetched via the K8s API.

Thanks!

Answer 1

I ended up just using Spring's RestTemplate to query prometheus directly. I'm only working against a dev cluster, so I just leveraged Len's install of prometheus, which means the endpoint is something like: http://prometheus.lens-metrics:9090

Here's their query, which I've reused below: https://github.com/lensapp/lens/blob/652319ac89cd30ace570539cb74be081da24781d/src/main/prometheus/lens.ts#L122

String query = String
    .format(
        "sum(kubelet_volume_stats_used_bytes{persistentvolumeclaim=\"%s\",namespace=\"%s\"}) by (persistentvolumeclaim, namespace)",
        pvc.getMetadata().getName(),
        pvc.getMetadata().getNamespace());

var response = restTemplateBuilder.build().getForEntity(prometheusEndpoint + "/api/v1/query?query={query}", String.class, query);
String json = response.getBody();

// E.g. {"status":"success","data":{"resultType":"vector","result":[{"metric":{"namespace":"luke",
//       "persistentvolumeclaim":"elasticsearch-data-es-ap-southeast-2c-0"},"value":[1622781271.612,"76865536"]}]}}
long bytesUsed = JsonPath.parse(json).read("data.result[0].value[1]", Long.class);