Skip to content

Event Streaming

Join our forum to discuss your ideas with Aiven community or check out our public roadmap.

Event Streaming

Categories

JUMP TO ANOTHER FORUM

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback

89 results found

  1. As a developer,
    I want error messages and logs to contain schema names and versions,
    so that I can quickly identify and troubleshoot issues related to specific schemas more efficiently.
    In addition, this improvement is very important when dealing with issues in referenced schemas because it provides more context in error messages and logs, making it easier to diagnose and resolve problems. This can significantly reduce the time spent on debugging and improve overall system maintainability.

    3 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    Pending Review  ·  0 comments  ·  Karapace  ·  Admin →
    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  2. As a developer,
    I want to have an enforced compatibility check on all existing schemas when the compatibility level is set to a more restrictive one (or for any change),
    so that I can ensure all schemas comply with the new restrictive compatibility level and maintain consistency in the schema registry.
    In addition, this improvement is important because it prevents potential issues when new schemas are registered or existing ones are updated, thereby increasing the reliability of the schema registry.

    3 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    Pending Review  ·  0 comments  ·  Karapace  ·  Admin →
    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  3. using prometheus endpoint at our kafka-service we need to have metrics from kafka with type .

    see below the metrics have notype - this should be set at the endpoint ,to ease the use of these metrics .
    see example

    TYPE kafkaservergroupcoordinatormetricsgroupcompletedrebalancecount untyped

    TYPE kafkaservergroupcoordinatormetricsoffsetcommit_rate untyped

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  4. As an organization (DevOps/Security/Vendor Manager) using Aiven Kafka,
    we want to determine the last connected cert status of Kafka users,
    so that we can know whether a kafka user certificate has been successfully updated.

    We automate certificate rolling to an extent with terraform. Different teams of devs generally own their section of terraform creating Kafka users. As of right now every two years those certs expire, and clicking the "Yes I've updated" in the Aiven console just silences the alert, and provides no real time verification from the running kafka that a certificate has been updated.

    This means a user…

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  5. As a Kafka user
    I want to be able to have consumers fetch from the closest replica (KIP-392)
    so that I can reduce Inter-AZ costs
    In addition, this will also reduce latency of consumer calls.

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  6. As data streaming architect
    I want to be able to export records from Kafka to GCS and use values in the record to define the bucket or file name
    so that I can organize data by those values to make them easier to find and process.

    Use case is a multi-user/multi-tenant application where user info is a value in the record. Need to be able to organize the output in object storage by that value somehow.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  7. As a developer / DevOps
    I want to be able to Sink CDC data into Apache Iceberg format
    so that I can analyze data using time travel feature of AWS Athena
    In addition, we may find a way for the current "Aiven - Amazon AWS S3 Sink" connector to be able to produce "Apache Iceberg" in addition of "Parquet" format or we may provide a dedicated connector like the one from this repository : https://github.com/tabular-io/iceberg-kafka-connect

    Yours faithfully,
    LCDP

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  8. As a managed Kafka administrator
    I want to have the ability to black-list a bunch of IP addresses I believe are suspicious from my Kafka service, so that I can prevent my service being disrupted by unexpected traffic
    In addition, I would like to keep the current allow-list to be able to allow know IPs and a way to resolve a conflict between allow and block list where block list takes priority.

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  9. As application engg,
    I want to develop a csv connector where data is ingested from flat files (CSV) and create a stream of records that can be processed in Apache Kafka. It is similar to what confluent kafka provides here (https://docs.confluent.io/kafka-connectors/spooldir/current/connectors/csv_source_connector.html)

    4 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  10. As an OPS Engineer
    I want to be able to balance the cluster based on CPU usage rather than Disk usage. The current algorithm focuses on Disk usage which is not optimum for our application.
    Can we have an option to apply partition rebalancing based on CPU usage?

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  11. As a platform engineer
    I want the service updates to be versioned and to be able to select a specific version (Kafka, Karapace, etc) to update to so that I can perform correct change promotion from non-prod to production (instead of being forced to always apply the latest) and so that if a new version is released while an update is in-flight it would not lead to different versions running across the brokers (as is currently the case, which we have experienced can lead to incompatibilities). Additionally, I would like it to be clear what version(s) is currently running in…

    12 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  12. As IT Architect
    I want to provide complete traceability within my microservice mesh where 2 microservice communicate via Outbox pattern via Debezium connector. In order to achieve this, Debezium connector requires some of the OpenTelemtry APIs are on the Kafka Connect classpath.
    I want to have this feature so that I can see complete chain of interactions for specific request, observe what and where time is spent inside particular microservice and find out possible bottlenecks.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  13. As an AIven developer user
    I want to use the Java code generated from Aiven API to communicate with an Aiven managed KafkaConnect cluster
    so that I can build micro services in Java platform to manage various connectors on the cluster
    In addition, this gives more security for Aiven users and data while the standard REST API supports only BasicAuth.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  14. As an ops engineer,
    I want to be able to declare fined grained ACLs
    so that I avoid having to grant "admin" rights to users which only need "DeleteRecords" rights on specific topics.
    The client uses Kafka Streams, which needs specific rights (https://docs.confluent.io/platform/current/streams/developer-guide/security.html#required-acl-setting-for-secure-ak-clusters) which are not covered by aiven predefined rights.
    Currently, the "admin" is to broad for such access (I don't want the user to be able to create topics).

    8 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  15. As a Kafka Administrator
    I want to ahve ability to scalse Kafka clsuter without upgrade, that increses time of scale.
    In production this can become critical and lead to downtime. For example last 2 times the scale lasted for 12 hours.

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  16. As Developer
    I want to use Aiven's S3 sink connector and have the ability to control the size of output files (keep same size even if the traffic changes). In general, the connector has no lags and we want to flush the offset and write to file only when we have enough data. We use offset.flush.interval.ms for it but when the traffic increases, the amount of data arrives in the configured interval is increased and can cause an OOM issue. In addition, when we pause the connector for couple of minutes and we gather a lag, it can also lead…

    5 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  17. As a developer
    I want to have the ability to use our own S3 bucket for storing Kafka tiered storage
    so that I can access the data from S3 and query some data for debugging (without streaming all the data to Kafka)

    3 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  18. As a data engineer
    I want Aiven Kafka Connect to offer the option to use the protocol buffer data format when serializíng the events that is sends to a Kafka broker.
    In my specific case, I need it to be possible in a Debezium connector for PostgreSQL.
    Additionally, it would be good to have as a user the option to define oneself the protobuf schema to use for serializing.

    5 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  19. Support RocksDB as a persistent data store for Flink

    As an engineer I need be able to run larger state Flink jobs in order to meet my data processing requirements

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  20. Debugging failed connectivity between Aiven for Apache MirrorMaker and an external Kafka configured using an integration endpoint is very difficult today. No errors are surfaced in the integration endpoint configuration screen itself, we must wait for the replication flow to attempt to start and then dig out cryptic errors from MM2 logs.

    Some kinds of errors (e.g. failure to build SSL keystores) are not even surfaced in MM2 logs and are only visible to Aiven operators.

    Please consider adding a basic connectivity check to allow for quicker troubleshooting and iteration. This check should ensure that the network path between Aiven…

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
← Previous 1 3 4 5
  • Don't see your idea?