Skip to content

Event Streaming

Join our forum to discuss your ideas with Aiven community or check out our public roadmap.

Event Streaming

Categories

JUMP TO ANOTHER FORUM

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback

95 results found

  1. Debugging failed connectivity between Aiven for Apache MirrorMaker and an external Kafka configured using an integration endpoint is very difficult today. No errors are surfaced in the integration endpoint configuration screen itself, we must wait for the replication flow to attempt to start and then dig out cryptic errors from MM2 logs.

    Some kinds of errors (e.g. failure to build SSL keystores) are not even surfaced in MM2 logs and are only visible to Aiven operators.

    Please consider adding a basic connectivity check to allow for quicker troubleshooting and iteration. This check should ensure that the network path between Aiven…

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  2. As a developer,
    I want to have an easy setup of my Kafka connectors when they are using internal Aiven services such as Postgres or OpenSearch
    so that I can save time, avoid mistakes, and enforce the strength of the platform.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  3. As an architect, I want to bring business logic out of the database and into a decoupled stream processing / event driven architecture framework. With SQL Server, I want to stream changes to Apache Kafka, using Debezium. This must support use cases where sensitivity classifications in SQL Server are used/required for things like PII.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  4. As a developer, I need to be able to read and write from Databricks so that I can complement my existing data architecture

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  5. As a developer, I need Flink to be able to automatically increase compute resources to a self-defined threshold to seamlessly process significant increases in traffic in order to maintain performance. This increase in traffic would be for either a permanently higher workload, or for a temporary pre-defined amount of time, and would be triggered by a customer request, with subsequent additional billing impacts

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  6. As a Cloud platform engineer, I need to have a view of Kafka cluster resources consumption across all clients IDs. Such information is needed to configure quotas and to manage them.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  7. As a kafka user, I want to configure the threshold used to send CPU warning emails. Right now we are getting emails because the CPU exceeds 50%, which is irrelevant. We know we can configure the emails to go to another mail address but in that case we would loose all other technical mails as well.
    It would also be good for us if we can turn off that single alert.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  8. If an external Kafka service has brokers with hostnames that are not publicly resolvable, for example, where hostnames are registered in a corporate DNS server which is not accessible from outside, it is currently not possible for MM2 to successfully connect.

    The workaround to use IP addresses is not sufficient, since after bootstrap the broker hostnames will be returned and subsequent connections will go there.

    We will also likely see issues with TLS handshakes when brokers present certificates belonging to hostnames that the Aiven side does not understand.

    The only current workaround is for the configuration of the external Kafka…

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  9. As a developer / operator,
    I want to see Kafka topics grouped by type in Aiven Console
    so that I can distinguish between topics created by Schema Registry, MirrorMaker2, Connect and directly in Kafka itself.
    In addition, i want to apply filters and sorting by type.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  10. As Software Engineer at Wex
    I want to send additional metrics to Datadog
    so that I can send important metrics from Aiven's cluster as these ones:
    IsrShrinksPerSec
    IsrExpandsPerSec
    ActiveControllerCount
    OfflinePartitionsCount
    TotalTimeMs
    PurgatorySize
    RequestsPerSec
    Network bytes sent/received
    BytesInPerSec/BytesOutPerSec

    According to Datadog's documentation, these metrics are considered highly significant.
    https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/#kafka-emitted-metrics

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  11. As a developer,
    I want to be able to select password length and difficulty when creating a service user,
    so that I can stay complaint with internal security and compliance rules.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  12. As a developer,
    I want to be notified when I have stale credentials in my connectors,
    so that I can update them and make sure connection from a connector to a source or sink system could be successfully established.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  13. As a developer,
    I want to connect to my Cassandra databases,
    so that I can use Change Data Capture (CDC) pattern to be notified about data changes and can ingest that data in to my data streaming pipeline for further use and processing.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  14. As an architect
    I want to have a message bus solution that is cost effective while still maintaining level of service for rare high load scenarios
    so that I can save money, simplify operations, and align value to usage

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  15. As Data Platform Principal Engineer
    I want to use emails as Kafka usernames when OAuth authentication is configured
    so that I can use Databricks as SSO provider to reduce the amount of credentials that I need to manage and share with each user. This also improves security, because it automatically disable the access when someone leaves the company.

    In particular, when I am using a Databricks Service Principal for the authentication, it works as expected. The Databricks Service Principal is identified by an unique UUID. To make it working I have added a Kafka service user with that UUID as…

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  16. As SRE Engineer

    I want to customize ACL to allow kafka consumer operations to carry on, even while write lock gets triggered when the disk space reaches threshold limits of 95 or 97%. Given that the Kafka consumers' offset commits are relatively smaller, this option will not be detrimental.

    so that even when disk space reaches critical levels, it will not immediately impact consumer side operations

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  17. As a Solutions Architect
    I want to have consumer lag for kafka consumers available out of the box
    so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.

    Background
    1. there is a consumer lag on the default metrics dashboard but it does not work.
    2. I contact support and found that…

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  18. Add support for exactly-once delivery in Storage Write API for GBQ sink connector.

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  19. using prometheus endpoint at our kafka-service we need to have metrics from kafka with type .

    see below the metrics have notype - this should be set at the endpoint ,to ease the use of these metrics .
    see example

    TYPE kafkaservergroupcoordinatormetricsgroupcompletedrebalancecount untyped

    TYPE kafkaservergroupcoordinatormetricsoffsetcommit_rate untyped

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  20. As a managed Kafka administrator
    I want to have the ability to black-list a bunch of IP addresses I believe are suspicious from my Kafka service, so that I can prevent my service being disrupted by unexpected traffic
    In addition, I would like to keep the current allow-list to be able to allow know IPs and a way to resolve a conflict between allow and block list where block list takes priority.

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    How important is this to you?

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
  • Don't see your idea?