kafka_connect_connector_metrics availability over Prometheus
Our customer Jago wants to monitor the status of connectors and tasks but currently can't find relevant metrics to do so. They want to be able to monitor the status of connectors and tasks on a dashboard and also get notified whenever a connector has not been running for X minutes.
Jago has a connector running but can not find the metrics for kafka.connect:type=connector-metrics,connector=*.
The specific metrics they are looking for is the one related to the status of a connector. For example, in the customers current self-managed kafka connect, they have the following metrics. This is convenient because they can get notified whenever a connector is not running.
kafkaconnectconnector_metrics{status="running",connector="lfs-source-kafka-notifications",} 1.0
They are also facing a similar issue with task status as they don't see metrics for it, while on their current cluster has:
kafkaconnectconnector_status{connector="lfs-source-kafka-notification-created",status="running",task="0",} 1.0
They have shared the attached metrics txt related to that connector that they got from prometheus.
-
Slack discussion: https://aiven-io.slack.com/archives/C01TN5ZQ9MZ/p1705552348670449