67 results found
-
Support for "Apache Iceberg" format while sinking CDC
As a developer / DevOps
I want to be able to Sink CDC data into Apache Iceberg format
so that I can analyze data using time travel feature of AWS Athena
In addition, we may find a way for the current "Aiven - Amazon AWS S3 Sink" connector to be able to produce "Apache Iceberg" in addition of "Parquet" format or we may provide a dedicated connector like the one from this repository : https://github.com/tabular-io/iceberg-kafka-connectYours faithfully,
LCDP8 votes -
Support for "stretched" cluster across multi regions/clouds for Aiven for Apache Kafka
As a developer / SRE,
I want to have my Apache Kafka cluster to run across multiple regions and potentially even clouds,
so that I can ensure high availability setup and have near zero RTO and RPO in case of DR, i.e. network, region or provider failure.
In addition, I want to benefit from the geo-distributed cluster setup and read/write from/to the geographically closest broker to optimise my network latency and cost.55 votes -
Allow inspected custom code (JARs) in Apache Flink and Apache Kafka Connect
As a developer,
I want to be able to upload my custom code (JARs),
so that I can use it to build sophisticated or non-standard use-cases in Flink and Kafka Connect,
In addition, I can keep custom implementation private.52 votes -
Support Grouping of OAuth2/OIDC Users
As a platform engineer
I want to group multiple users based on their role (OAuth2/OIDC claim)
so that I can reduce the number of required Kafka users and ACL entries that need to be managed.Currently, every user / identity connecting via OAuth2/OIDC has a 1:1 mapping to a Kafka user (the username is taken from the sub claim). This is cumbersome and leads to significant overhead if for example multiple identities / users with the same permissions want to access the Kafka service. Kafka users and ACLs need to be created for every single identity, even though they share…
2 votes -
Monitoring consume lag for kafka not out of the box
As a Solutions Architect
I want to have consumer lag for kafka consumers available out of the box
so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.Background
1. there is a consumer lag on the default metrics dashboard but it does not work.
2. I contact support and found that…1 vote -
Kafka Connect GCS Sink: Support using field values to define bucket name or file name prefix
As data streaming architect
I want to be able to export records from Kafka to GCS and use values in the record to define the bucket or file name
so that I can organize data by those values to make them easier to find and process.Use case is a multi-user/multi-tenant application where user info is a value in the record. Need to be able to organize the output in object storage by that value somehow.
3 votes -
Aiven's S3 sink connector - Support configure offset flush max size
As Developer
I want to use Aiven's S3 sink connector and have the ability to control the size of output files (keep same size even if the traffic changes). In general, the connector has no lags and we want to flush the offset and write to file only when we have enough data. We useoffset.flush.interval.ms
for it but when the traffic increases, the amount of data arrives in the configured interval is increased and can cause an OOM issue. In addition, when we pause the connector for couple of minutes and we gather a lag, it can also lead…5 votes -
fetch closest replica
As a Kafka user
I want to be able to have consumers fetch from the closest replica (KIP-392)
so that I can reduce Inter-AZ costs
In addition, this will also reduce latency of consumer calls.2 votes -
Support for Protobuf serialization of Events (and Keys) in Aiven Kafka Connnect
As a data engineer
I want Aiven Kafka Connect to offer the option to use the protocol buffer data format when serializíng the events that is sends to a Kafka broker.
In my specific case, I need it to be possible in a Debezium connector for PostgreSQL.
Additionally, it would be good to have as a user the option to define oneself the protobuf schema to use for serializing.5 votes -
Schema Validation on Apache Kafka broker side
As a developer,
I want to make sure schema is being validated not just on client but also on a broker side,
so that I can make sure all messages in the topic correspond to the same schema and does not contain any mixed schemas.18 votes -
Karapace Schema Registry certificate authentication
As a developer I should be able to authenticate with same service user certificate for both Kafka and Schema Registry
5 votes -
Programmatic Apache Kafka Consumer (group) management
As a developer,
I want to programmatically manage my consumers and consumer groups,
so that I can see the status of them, perform CRUD operations, show members of a group, reset offsets and similar.
In addition, I want to be able to do so also in Aiven Console.19 votes -
Support for public CA for SASL for Aiven for Apache Kafka
As developer,
I want to use public CA (i.e. Let's Encrypt),
so that I can connect to my Apache for Kafka service without installing any additional certificates.
In addition, I can trust publish authority issuing certificates instead of validating 3rd party certificates.15 votes -
Add OpenTelemetry API and SDK to enable traceability in Debezium connector
As IT Architect
I want to provide complete traceability within my microservice mesh where 2 microservice communicate via Outbox pattern via Debezium connector. In order to achieve this, Debezium connector requires some of the OpenTelemtry APIs are on the Kafka Connect classpath.
I want to have this feature so that I can see complete chain of interactions for specific request, observe what and where time is spent inside particular microservice and find out possible bottlenecks.2 votes -
Aiven API support for Kafka Connect Java clients
As an AIven developer user
I want to use the Java code generated from Aiven API to communicate with an Aiven managed KafkaConnect cluster
so that I can build micro services in Java platform to manage various connectors on the cluster
In addition, this gives more security for Aiven users and data while the standard REST API supports only BasicAuth.2 votes -
Move Karapace updates out of maintenance updates into IaC definitions
As a customer
I want to be able to select Karapace version updates in Terraform code instead of maintenance updates GUI
so that I can control when Karapace updates are applied from my IaC definition.
In addition, Karapace updates can introduce service interruptions for users because all consumer instances are terminated. Moving it out of maintenance updates means less user interruptions on often more important maintenance updates.4 votes -
Support Databricks driver for Apache Kafka Connect JDBC sink connector
As a developer,
I want to have support for Databricks driver for Apache Kafka Connect JDBC sink connector,
so that I can write data from Aiven for Apache Kafka to Databricks Spark for further processing, analysis and consumption.11 votes -
Audit logging as self-service option
As a developer,
I want to enable audit logging for my service,
so that I can keep track of breadcrumbs left by all the changes made to a service to ensure compliance.11 votes -
Configurable maximum validity for service user certificates
As a developer / security engineer,
I want to be able to define lifetime of certificates (i.e. 30-90 days) used to authenticate my service users,
so that I can have certificate rotation policies in place to ensure compliance and security best practices.10 votes -
Karapace OAuth2 support
As a developer / security engineer,
I want to be able to authenticate (i.e. Azure AD) and authorize (OAuth) my service users using OAuth2,
so that I can manage all users in a central place and don't need to worry about missing something during on- and off- boarding.9 votesGathering Interest · AdminMichael Tansini (Product Manager Data Streaming (Kafka and Flink), Aiven.io) responded
- Don't see your idea?