95 results found
-
Connectivity check for MirrorMaker with external Kafka integration
Debugging failed connectivity between Aiven for Apache MirrorMaker and an external Kafka configured using an integration endpoint is very difficult today. No errors are surfaced in the integration endpoint configuration screen itself, we must wait for the replication flow to attempt to start and then dig out cryptic errors from MM2 logs.
Some kinds of errors (e.g. failure to build SSL keystores) are not even surfaced in MM2 logs and are only visible to Aiven operators.
Please consider adding a basic connectivity check to allow for quicker troubleshooting and iteration. This check should ensure that the network path between Aiven…
2 votes -
Ease the setup of Kafka Connectors when using internal Aiven services
As a developer,
I want to have an easy setup of my Kafka connectors when they are using internal Aiven services such as Postgres or OpenSearch
so that I can save time, avoid mistakes, and enforce the strength of the platform.2 votes -
Improve support for Debezium SQL Server use cases
As an architect, I want to bring business logic out of the database and into a decoupled stream processing / event driven architecture framework. With SQL Server, I want to stream changes to Apache Kafka, using Debezium. This must support use cases where sensitivity classifications in SQL Server are used/required for things like PII.
2 votes -
Add external Delta Lake connector for Apache Flink
As a developer, I need to be able to read and write from Databricks so that I can complement my existing data architecture
2 votes -
Auto-Scaling
As a developer, I need Flink to be able to automatically increase compute resources to a self-defined threshold to seamlessly process significant increases in traffic in order to maintain performance. This increase in traffic would be for either a permanently higher workload, or for a temporary pre-defined amount of time, and would be triggered by a customer request, with subsequent additional billing impacts
2 votes -
Visualization of client IDs that interact with a Kafka cluster
As a Cloud platform engineer, I need to have a view of Kafka cluster resources consumption across all clients IDs. Such information is needed to configure quotas and to manage them.
2 votes -
Configure CPU threshold for alert mails in kafka
As a kafka user, I want to configure the threshold used to send CPU warning emails. Right now we are getting emails because the CPU exceeds 50%, which is irrelevant. We know we can configure the emails to go to another mail address but in that case we would loose all other technical mails as well.
It would also be good for us if we can turn off that single alert.2 votes -
MirrorMaker 2 connectivity for Kafka Brokers with private or internal hostnames
If an external Kafka service has brokers with hostnames that are not publicly resolvable, for example, where hostnames are registered in a corporate DNS server which is not accessible from outside, it is currently not possible for MM2 to successfully connect.
The workaround to use IP addresses is not sufficient, since after bootstrap the broker hostnames will be returned and subsequent connections will go there.
We will also likely see issues with TLS handshakes when brokers present certificates belonging to hostnames that the Aiven side does not understand.
The only current workaround is for the configuration of the external Kafka…
2 votes -
Group topics by type when displayed in Aiven Console
As a developer / operator,
I want to see Kafka topics grouped by type in Aiven Console
so that I can distinguish between topics created by Schema Registry, MirrorMaker2, Connect and directly in Kafka itself.
In addition, i want to apply filters and sorting by type.2 votes -
Sending Apache Kafka metrics to Datadog
As Software Engineer at Wex
I want to send additional metrics to Datadog
so that I can send important metrics from Aiven's cluster as these ones:
IsrShrinksPerSec
IsrExpandsPerSec
ActiveControllerCount
OfflinePartitionsCount
TotalTimeMs
PurgatorySize
RequestsPerSec
Network bytes sent/received
BytesInPerSec/BytesOutPerSecAccording to Datadog's documentation, these metrics are considered highly significant.
https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/#kafka-emitted-metrics2 votes -
Ability to choose Apache Kafka service user password length using Aiven Console
As a developer,
I want to be able to select password length and difficulty when creating a service user,
so that I can stay complaint with internal security and compliance rules.2 votes -
Send a user alert in case stale credentials are being detected in connector configuration for Aiven for Apache Kafka Connect.
As a developer,
I want to be notified when I have stale credentials in my connectors,
so that I can update them and make sure connection from a connector to a source or sink system could be successfully established.2 votes -
Debezium Connector for Cassandra
As a developer,
I want to connect to my Cassandra databases,
so that I can use Change Data Capture (CDC) pattern to be notified about data changes and can ingest that data in to my data streaming pipeline for further use and processing.2 votes -
Serverless option for Aiven for Kafka
As an architect
I want to have a message bus solution that is cost effective while still maintaining level of service for rare high load scenarios
so that I can save money, simplify operations, and align value to usage1 vote -
Emails should be valid Kafka usernames with OAuth
As Data Platform Principal Engineer
I want to use emails as Kafka usernames when OAuth authentication is configured
so that I can use Databricks as SSO provider to reduce the amount of credentials that I need to manage and share with each user. This also improves security, because it automatically disable the access when someone leaves the company.In particular, when I am using a Databricks Service Principal for the authentication, it works as expected. The Databricks Service Principal is identified by an unique UUID. To make it working I have added a Kafka service user with that UUID as…
1 vote -
ACL
As SRE Engineer
I want to customize ACL to allow kafka consumer operations to carry on, even while write lock gets triggered when the disk space reaches threshold limits of 95 or 97%. Given that the Kafka consumers' offset commits are relatively smaller, this option will not be detrimental.
so that even when disk space reaches critical levels, it will not immediately impact consumer side operations
1 vote -
Monitoring consume lag for kafka not out of the box
As a Solutions Architect
I want to have consumer lag for kafka consumers available out of the box
so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.Background
1. there is a consumer lag on the default metrics dashboard but it does not work.
2. I contact support and found that…1 vote -
Exactly-Once support in Storage Write API from our GBQ sink connector
Add support for exactly-once delivery in Storage Write API for GBQ sink connector.
1 vote -
untyped metrics from prometheus endpoint
using prometheus endpoint at our kafka-service we need to have metrics from kafka with type .
see below the metrics have notype - this should be set at the endpoint ,to ease the use of these metrics .
see exampleTYPE kafkaservergroupcoordinatormetricsgroupcompletedrebalancecount untyped
TYPE kafkaservergroupcoordinatormetricsoffsetcommit_rate untyped
1 vote -
Support for both allow and black lists in Kafka service
As a managed Kafka administrator
I want to have the ability to black-list a bunch of IP addresses I believe are suspicious from my Kafka service, so that I can prevent my service being disrupted by unexpected traffic
In addition, I would like to keep the current allow-list to be able to allow know IPs and a way to resolve a conflict between allow and block list where block list takes priority.1 voteat the moment we recommend using our current networking whitelisting capabilities
- Don't see your idea?