91 results found
-
Kafka versioned updates
As a platform engineer
I want the service updates to be versioned and to be able to select a specific version (Kafka, Karapace, etc) to update to so that I can perform correct change promotion from non-prod to production (instead of being forced to always apply the latest) and so that if a new version is released while an update is in-flight it would not lead to different versions running across the brokers (as is currently the case, which we have experienced can lead to incompatibilities). Additionally, I would like it to be clear what version(s) is currently running in…12 votes -
Support for "stretched" cluster across multi regions/clouds for Aiven for Apache Kafka
As a developer / SRE,
I want to have my Apache Kafka cluster to run across multiple regions and potentially even clouds,
so that I can ensure high availability setup and have near zero RTO and RPO in case of DR, i.e. network, region or provider failure.
In addition, I want to benefit from the geo-distributed cluster setup and read/write from/to the geographically closest broker to optimise my network latency and cost.52 votes -
Support for "Apache Iceberg" format while sinking CDC
As a developer / DevOps
I want to be able to Sink CDC data into Apache Iceberg format
so that I can analyze data using time travel feature of AWS Athena
In addition, we may find a way for the current "Aiven - Amazon AWS S3 Sink" connector to be able to produce "Apache Iceberg" in addition of "Parquet" format or we may provide a dedicated connector like the one from this repository : https://github.com/tabular-io/iceberg-kafka-connectYours faithfully,
LCDP6 votes -
Kafka fined grained ACLs
As an ops engineer,
I want to be able to declare fined grained ACLs
so that I avoid having to grant "admin" rights to users which only need "DeleteRecords" rights on specific topics.
The client uses Kafka Streams, which needs specific rights (https://docs.confluent.io/platform/current/streams/developer-guide/security.html#required-acl-setting-for-secure-ak-clusters) which are not covered by aiven predefined rights.
Currently, the "admin" is to broad for such access (I don't want the user to be able to create topics).8 votes -
Allow inspected custom code (JARs) in Apache Flink and Apache Kafka Connect
As a developer,
I want to be able to upload my custom code (JARs),
so that I can use it to build sophisticated or non-standard use-cases in Flink and Kafka Connect,
In addition, I can keep custom implementation private.50 votes -
Enhance error messages and logs with schema names and versions
As a developer,
I want error messages and logs to contain schema names and versions,
so that I can quickly identify and troubleshoot issues related to specific schemas more efficiently.
In addition, this improvement is very important when dealing with issues in referenced schemas because it provides more context in error messages and logs, making it easier to diagnose and resolve problems. This can significantly reduce the time spent on debugging and improve overall system maintainability.3 votes -
Enforce schema compatibility check on level change
As a developer,
I want to have an enforced compatibility check on all existing schemas when the compatibility level is set to a more restrictive one (or for any change),
so that I can ensure all schemas comply with the new restrictive compatibility level and maintain consistency in the schema registry.
In addition, this improvement is important because it prevents potential issues when new schemas are registered or existing ones are updated, thereby increasing the reliability of the schema registry.3 votes -
CSV kafka connector
As application engg,
I want to develop a csv connector where data is ingested from flat files (CSV) and create a stream of records that can be processed in Apache Kafka. It is similar to what confluent kafka provides here (https://docs.confluent.io/kafka-connectors/spooldir/current/connectors/csv_source_connector.html)4 votes -
Aiven's S3 sink connector - Support configure offset flush max size
As Developer
I want to use Aiven's S3 sink connector and have the ability to control the size of output files (keep same size even if the traffic changes). In general, the connector has no lags and we want to flush the offset and write to file only when we have enough data. We useoffset.flush.interval.msfor it but when the traffic increases, the amount of data arrives in the configured interval is increased and can cause an OOM issue. In addition, when we pause the connector for couple of minutes and we gather a lag, it can also lead…5 votes -
Certificate file support for Kafka connectors
As a developer / security engineer,
I want to be able to authenticate my Apache Kafka connectors via mTLS,
so that I connect to my external services in a secure way.24 votesRoadmapped ·AdminMichael Tansini (Product Manager Data Streaming (Kafka and Flink), Aiven.io) responded
-
Secure secrets in Hashicorp Vault and use them in Kafka Connect.
As DevOps/SecOps engineer,
I want to securely manage my credentials in Kafka Connect connectors,
so that I can make sure no credentials are being leaked in connector configuration.
In addition, I can manage my credentials in Hashicorp Vault and exercise credential rotation without changing my connector configuration.26 votes -
Support for Protobuf serialization of Events (and Keys) in Aiven Kafka Connnect
As a data engineer
I want Aiven Kafka Connect to offer the option to use the protocol buffer data format when serializíng the events that is sends to a Kafka broker.
In my specific case, I need it to be possible in a Debezium connector for PostgreSQL.
Additionally, it would be good to have as a user the option to define oneself the protobuf schema to use for serializing.5 votes -
Kafka Connect GCS Sink: Support using field values to define bucket name or file name prefix
As data streaming architect
I want to be able to export records from Kafka to GCS and use values in the record to define the bucket or file name
so that I can organize data by those values to make them easier to find and process.Use case is a multi-user/multi-tenant application where user info is a value in the record. Need to be able to organize the output in object storage by that value somehow.
2 votes -
Provide documentation for Karapace REST API
As a developer
I want to use a REST API against my Kafka instance
so that I can write simple scripts without using client libraries.It doesn't seem like there's comprehensive API documentation for what endpoints and functionality are supported by the Karapace REST API. The website says it's a drop-in replacement for the Kafka REST API proxy but unless that comes with guarantees that it'll stay up-to-date with any changes in the Confluent Kafka REST API proxy, it's hard to trust that. Some users may find it preferable to just have the documentation for Karapace's endpoints.
1 vote -
Exactly-Once support in Storage Write API from our GBQ sink connector
Add support for exactly-once delivery in Storage Write API for GBQ sink connector.
1 vote -
Schema Validation on Apache Kafka broker side
As a developer,
I want to make sure schema is being validated not just on client but also on a broker side,
so that I can make sure all messages in the topic correspond to the same schema and does not contain any mixed schemas.18 votes -
Run Karapace as dedicated managed service
As a developer,
I want to run Karapace as fully managed dedicated service,
so that I can use it with Apache Kafka running on and outside of Aiven.
In addition, I can use same Karapace service against multiple Apache Kafka services.17 votes -
Programmatic Apache Kafka Consumer (group) management
As a developer,
I want to programmatically manage my consumers and consumer groups,
so that I can see the status of them, perform CRUD operations, show members of a group, reset offsets and similar.
In addition, I want to be able to do so also in Aiven Console.18 votes -
untyped metrics from prometheus endpoint
using prometheus endpoint at our kafka-service we need to have metrics from kafka with type .
see below the metrics have notype - this should be set at the endpoint ,to ease the use of these metrics .
see exampleTYPE kafkaservergroupcoordinatormetricsgroupcompletedrebalancecount untyped
TYPE kafkaservergroupcoordinatormetricsoffsetcommit_rate untyped
1 vote -
Karapace Schema Registry certificate authentication
As a developer I should be able to authenticate with same service user certificate for both Kafka and Schema Registry
4 votes
- Don't see your idea?