38 results found
-
Create a Backup to Azure Blob Storage for Local Region Restore - DR
As an application owner,
I want to be able to store data in Blob Storage for local recover from an outage using the backups on Blob storage and also be able to restore accidentally dropped topics. This backup would potentially include hundreds of topics.2 votes -
Support Grouping of OAuth2/OIDC Users
As a platform engineer
I want to group multiple users based on their role (OAuth2/OIDC claim)
so that I can reduce the number of required Kafka users and ACL entries that need to be managed.Currently, every user / identity connecting via OAuth2/OIDC has a 1:1 mapping to a Kafka user (the username is taken from the sub claim). This is cumbersome and leads to significant overhead if for example multiple identities / users with the same permissions want to access the Kafka service. Kafka users and ACLs need to be created for every single identity, even though they share…
2 votes -
fetch closest replica
As a Kafka user
I want to be able to have consumers fetch from the closest replica (KIP-392)
so that I can reduce Inter-AZ costs
In addition, this will also reduce latency of consumer calls.2 votes -
Configure CPU threshold for alert mails in kafka
As a kafka user, I want to configure the threshold used to send CPU warning emails. Right now we are getting emails because the CPU exceeds 50%, which is irrelevant. We know we can configure the emails to go to another mail address but in that case we would loose all other technical mails as well.
It would also be good for us if we can turn off that single alert.2 votes -
Group topics by type when displayed in Aiven Console
As a developer / operator,
I want to see Kafka topics grouped by type in Aiven Console
so that I can distinguish between topics created by Schema Registry, MirrorMaker2, Connect and directly in Kafka itself.
In addition, i want to apply filters and sorting by type.2 votes -
Sending Apache Kafka metrics to Datadog
As Software Engineer at Wex
I want to send additional metrics to Datadog
so that I can send important metrics from Aiven's cluster as these ones:
IsrShrinksPerSec
IsrExpandsPerSec
ActiveControllerCount
OfflinePartitionsCount
TotalTimeMs
PurgatorySize
RequestsPerSec
Network bytes sent/received
BytesInPerSec/BytesOutPerSecAccording to Datadog's documentation, these metrics are considered highly significant.
https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/#kafka-emitted-metrics2 votes -
Ability to choose Apache Kafka service user password length using Aiven Console
As a developer,
I want to be able to select password length and difficulty when creating a service user,
so that I can stay complaint with internal security and compliance rules.2 votes -
ACL
As SRE Engineer
I want to customize ACL to allow kafka consumer operations to carry on, even while write lock gets triggered when the disk space reaches threshold limits of 95 or 97%. Given that the Kafka consumers' offset commits are relatively smaller, this option will not be detrimental.
so that even when disk space reaches critical levels, it will not immediately impact consumer side operations
1 vote -
Monitoring consume lag for kafka not out of the box
As a Solutions Architect
I want to have consumer lag for kafka consumers available out of the box
so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.Background
1. there is a consumer lag on the default metrics dashboard but it does not work.
2. I contact support and found that…1 vote -
untyped metrics from prometheus endpoint
using prometheus endpoint at our kafka-service we need to have metrics from kafka with type .
see below the metrics have notype - this should be set at the endpoint ,to ease the use of these metrics .
see exampleTYPE kafkaservergroupcoordinatormetricsgroupcompletedrebalancecount untyped
TYPE kafkaservergroupcoordinatormetricsoffsetcommit_rate untyped
1 vote -
"Last Used" field on Kafka Certificates in Console/API
As an organization (DevOps/Security/Vendor Manager) using Aiven Kafka,
we want to determine the last connected cert status of Kafka users,
so that we can know whether a kafka user certificate has been successfully updated.We automate certificate rolling to an extent with terraform. Different teams of devs generally own their section of terraform creating Kafka users. As of right now every two years those certs expire, and clicking the "Yes I've updated" in the Aiven console just silences the alert, and provides no real time verification from the running kafka that a certificate has been updated.
This means a user…
1 vote -
Support for both allow and black lists in Kafka service
As a managed Kafka administrator
I want to have the ability to black-list a bunch of IP addresses I believe are suspicious from my Kafka service, so that I can prevent my service being disrupted by unexpected traffic
In addition, I would like to keep the current allow-list to be able to allow know IPs and a way to resolve a conflict between allow and block list where block list takes priority.1 voteShelved ·AdminMichael Tansini (Product Manager Data Streaming (Kafka and Flink), Aiven.io) responded
at the moment we recommend using our current networking whitelisting capabilities
-
Cluster leader balancing CPU vs Disk - can we choose?
As an OPS Engineer
I want to be able to balance the cluster based on CPU usage rather than Disk usage. The current algorithm focuses on Disk usage which is not optimum for our application.
Can we have an option to apply partition rebalancing based on CPU usage?1 vote -
Ability to scale Kafka cluster without upgrade
As a Kafka Administrator
I want to ahve ability to scalse Kafka clsuter without upgrade, that increses time of scale.
In production this can become critical and lead to downtime. For example last 2 times the scale lasted for 12 hours.1 vote -
kafka_connect_connector_metrics availability over Prometheus
Our customer Jago wants to monitor the status of connectors and tasks but currently can't find relevant metrics to do so. They want to be able to monitor the status of connectors and tasks on a dashboard and also get notified whenever a connector has not been running for X minutes.
Jago has a connector running but can not find the metrics for kafka.connect:type=connector-metrics,connector=*.
The specific metrics they are looking for is the one related to the status of a connector. For example, in the customers current self-managed kafka connect, they have the following metrics. This is convenient because they…
1 vote -
Automated dynamic quota configuration
As a Cloud platform engineer, I need to have an automated way to set up and update quota configurations on a cluster, taking into account changes in resources consumption patterns amongst producers and consumers.
1 vote -
Visualization of client IDs that interact with a Kafka cluster
As a Cloud platform engineer, I need to have a view of Kafka cluster resources consumption across all clients IDs. Such information is needed to configure quotas and to manage them.
1 vote -
Expose MirrorMaker 2 replication.policy.separator property
As a devops developer
I want to change the replication policy separator
so that I can use our existing topic separators without conflicting with MM2.
In addition, I find it extremely inconvenient to have to change our existing schema because we cannot configure the replication.policy.separator in MM2.1 voteWe can see it as useful configuration to expose. At the same time it will need to gather a bit more interest to be taken in to development.
Thanks!
- Don't see your idea?