Skip to content

Telemetry Client Service

ETSI HypO is deployed in a central management cluster alongside with a Prometheus service used to monitor the state of this cluster. ETSI HypO leverages this central Prometheus service to gather telemetry data across all managed clusters, where ETSI HypO deploys user services.

To do so, ETSI HypO requires the underlying OSS to create on-demand K8s clusters with a local cluster-wide Prometheus service in place. Given this, ETSI HypO uses a Telemetry Client service to configure the Prometheus instance on the managed cluster so as for the latter to publish service metrics to the central Prometheus instance. This way, all ETSI HypO-deployed services report their telemetry to a central Prometheus instance.

Upon the availability of a service's telemetry data, ETSI HypO visualizes the monitoring data that is aggregated at the central management cluster. This service provisions a Grafana service at the central management cluster and indicates certain service metrics that need to be visualized onto a Grafana dashboard. Then, ETSI HypO uses the Dashboard Client service to create new graphs on the central Grafana instance, based on the service metrics indicated as service characteristics in the Dashboard-aaS service.

End-user Service Flow

Telemetry receives a message from SONATA via the Kafka topic telemetry-create-prometheus-job-and-grafana-dashboard.

{
  "specversion": "1.0",
  "id": "4c813bc9-2276-44c1-ba91-af3d29d41dab",
  "source": "/process/serviceTelemetryProcess",
  "type": "telemetry-create-prometheus-job-and-grafana-dashboard",
  "time": "2024-10-30T11:28:34.161962096+02:00",
  "kogitoproctype": "BPMN",
  "kogitoprocinstanceid": "bb4c1652-7589-42bd-9417-f359c2f391ee",
  "kogitoprocist": "Active",
  "kogitoprocversion": "1.0",
  "kogitoprocid": "serviceTelemetryProcess",
  "data": {
    "serviceId": "ecc4345a-2097-4c3b-a40c-a9e791cdccbb",
    "namespaces": [
      "NAMESPACE_1",
      "NAMESPACE_2"
    ],
    "prometheus": {
      "host": "10.10.10.4",
      "port": 30180
    }
  }
}

Telemetry API then proceeds with retrieving the Kubernetes secret to update it by adding a new Prometheus job. A new job configuration is created using the serviceId as a job_name and a match for the namespaces provided by the incoming Kafka message. Then, a dashboard is created in Grafana under the folder ETSI HypO App Metrics for the corresponding namespaces.

Each dashboard contains 4 panels:

  • Memory Usage
  • CPU Usage
  • Network Received (The Network Received Mbits for the deployed app)
  • Network Transmitted (The Network Transmitted Mbits for the deployed app)

After the successful creation of the initial dashboard, a Loki dashboard is created under the folder: ETSI HypO End-User Service Logs.

Finally, Telemetry API sends a call back message to SONATA with the result of this operation via the Kafka topic sonata-service-dashboard-result. In the message all the relative dashboard URLs are included.

{
  "specversion": "1.0",
  "id": "21627e26-31eb-43e7-8343-92a696fd96b1",
  "source": "",
  "type": "sonata-service-dashboard-result",
  "kogitoprocrefid": "ac19c63c-307b-461f-8a0b-5cb9d2676c47",
  "data": {
    "dashboardId": "123",
    "serviceId": "ecc4345a-2097-4c3b-a40c-a9e791cdccbb",
    "metricsDashboardUrl": "http://localhost:3000/d/123",
    "logsDashboardUrl": "http://localhost:3000/d/456",
    "dashboardDataOrderStatus": "SUCCESS",
    "dashboardDataOrderMessage": "Telemetry was successful"
  }
}

Platform Service Flow

Telemetry API receives a message from SONATA via the Kafka topic telemetry-platform-service. Then a platform service dashboard is created in Grafana under the folder ETSI HypO Platform Service Logs.

Finally, a call back message towards SONATA with the result of this operation is sent via the Kafka topictelemetry-platform-service-result .

Delete Flow

Telemetry API receives a message via the Kafka topic telemetry-delete-prometheus-job-and-grafana-dashboard.

{
  "specversion": "1.0",
  "id": "898483a5-ebde-48c8-8a8a-2abd1bacacd5",
  "source": "/process/serviceTelemetryDeleteProcess",
  "type": "telemetry-delete-prometheus-job-and-grafana-dashboard",
  "time": "2024-11-11T14:31:44.365936393+02:00",
  "kogitoproctype": "BPMN",
  "kogitoprocinstanceid": "db641bde-d7d9-4ba0-87b7-2641a9aee28f",
  "kogitoprocist": "Active",
  "kogitoprocversion": "1.0",
  "kogitoprocid": "serviceTelemetryDeleteProcess",
  "data": {
    "serviceId": "ecc4345a-2097-4c3b-a40c-a9e791cdccbb"
  }
}

Telemetry API then deletes the Prometheus job that was created initially by using the serviceId as the job_name. After the successful update of the Kubernetes secret Telemetry, the Grafana dashboards that are associated with that serviceId are deleted.

Finally a callback message towards SONATA with the result of this operation is sent via Kafka topic sonata-service-delete-dashboard-result .

APIs

Telemetry Client listens to a Kafka topic where SONATA makes requests for telemetry endpoints' registration. When a such message is received, the Telemetry Client reads the input data from the Kafka topic and registers the corresponding metrics from a managed cluster to the central management cluster as shown in Table 1.

Table 1: Telemetry service API.
Northbound integration with Using Kafka topic Leads to southbound call Purpose
SONATA telemetry-create-prometheus-job-and-grafana-dashboard POST REST request to Kubernetes and Grafana create Prometheus job and Grafana dashboard
SONATA telemetry-platform-service POST REST request to Grafana to create ETSI HypO Platform Service Logs Loki Dashboard in Grafana
SONATA telemetry-delete-prometheus-job-and-grafana-dashboard DELETE REST request Grafana and Kubernetes edit secret to remove Prometheus job and delete Grafana dashboard
Southbound integration with Using Kafka topic Purpose
SONATA sonata-service-dashboard-result to inform about telemetry-create-prometheus-job-and-grafana-dashboard result
SONATA telemetry-platform-service-result to inform about telemetry-platform-service result
SONATA sonata-service-delete-dashboard-result to inform about telemetry-delete-prometheus-job-and-grafana-dashboard result

When telemetry data is available, the visualization of service metrics follows by the Telemetry Client.