67 results found

1. Add OpenTelemetry API and SDK to enable traceability in Debezium connector

   As IT Architect
   I want to provide complete traceability within my microservice mesh where 2 microservice communicate via Outbox pattern via Debezium connector. In order to achieve this, Debezium connector requires some of the OpenTelemtry APIs are on the Kafka Connect classpath.
   I want to have this feature so that I can see complete chain of interactions for specific request, observe what and where time is spent inside particular microservice and find out possible bottlenecks.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

2. Aiven API support for Kafka Connect Java clients

   As an AIven developer user
   I want to use the Java code generated from Aiven API to communicate with an Aiven managed KafkaConnect cluster
   so that I can build micro services in Java platform to manage various connectors on the cluster
   In addition, this gives more security for Aiven users and data while the standard REST API supports only BasicAuth.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

3. Support for RocksDB in Flink

   Support RocksDB as a persistent data store for Flink

   As an engineer I need be able to run larger state Flink jobs in order to meet my data processing requirements

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

4. Connectivity check for MirrorMaker with external Kafka integration

   Debugging failed connectivity between Aiven for Apache MirrorMaker and an external Kafka configured using an integration endpoint is very difficult today. No errors are surfaced in the integration endpoint configuration screen itself, we must wait for the replication flow to attempt to start and then dig out cryptic errors from MM2 logs.

   Some kinds of errors (e.g. failure to build SSL keystores) are not even surfaced in MM2 logs and are only visible to Aiven operators.

   Please consider adding a basic connectivity check to allow for quicker troubleshooting and iteration. This check should ensure that the network path between Aiven and the external Kafka is correct and the chosen authentication method and credentials are valid.

   Debugging failed connectivity between Aiven for Apache MirrorMaker and an external Kafka configured using an integration endpoint is very difficult today. No errors are surfaced in the integration endpoint configuration screen itself, we must wait for the replication flow to attempt to start and then dig out cryptic errors from MM2 logs.

   Some kinds of errors (e.g. failure to build SSL keystores) are not even surfaced in MM2 logs and are only visible to Aiven operators.

   Please consider adding a basic connectivity check to allow for quicker troubleshooting and iteration. This check should ensure that the network path between Aiven… more

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® MirrorMaker 2  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

5. Ease the setup of Kafka Connectors when using internal Aiven services

   As a developer,
   I want to have an easy setup of my Kafka connectors when they are using internal Aiven services such as Postgres or OpenSearch
   so that I can save time, avoid mistakes, and enforce the strength of the platform.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

6. Add external Delta Lake connector for Apache Flink

   As a developer, I need to be able to read and write from Databricks so that I can complement my existing data architecture

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

7. Auto-Scaling

   As a developer, I need Flink to be able to automatically increase compute resources to a self-defined threshold to seamlessly process significant increases in traffic in order to maintain performance. This increase in traffic would be for either a permanently higher workload, or for a temporary pre-defined amount of time, and would be triggered by a customer request, with subsequent additional billing impacts

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

8. Configure CPU threshold for alert mails in kafka

   As a kafka user, I want to configure the threshold used to send CPU warning emails. Right now we are getting emails because the CPU exceeds 50%, which is irrelevant. We know we can configure the emails to go to another mail address but in that case we would loose all other technical mails as well.
   It would also be good for us if we can turn off that single alert.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

9. MirrorMaker 2 connectivity for Kafka Brokers with private or internal hostnames

   If an external Kafka service has brokers with hostnames that are not publicly resolvable, for example, where hostnames are registered in a corporate DNS server which is not accessible from outside, it is currently not possible for MM2 to successfully connect.

   The workaround to use IP addresses is not sufficient, since after bootstrap the broker hostnames will be returned and subsequent connections will go there.

   We will also likely see issues with TLS handshakes when brokers present certificates belonging to hostnames that the Aiven side does not understand.

   The only current workaround is for the configuration of the external Kafka service to change so that extra listeners or certificates are added, but this is often very difficult to achieve in a production environment where change is tightly managed or where downtime is to be avoided.

   If an external Kafka service has brokers with hostnames that are not publicly resolvable, for example, where hostnames are registered in a corporate DNS server which is not accessible from outside, it is currently not possible for MM2 to successfully connect.

   The workaround to use IP addresses is not sufficient, since after bootstrap the broker hostnames will be returned and subsequent connections will go there.

   We will also likely see issues with TLS handshakes when brokers present certificates belonging to hostnames that the Aiven side does not understand.

   The only current workaround is for the configuration of the external Kafka… more

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® MirrorMaker 2  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

10. Group topics by type when displayed in Aiven Console

    As a developer / operator,
    I want to see Kafka topics grouped by type in Aiven Console
    so that I can distinguish between topics created by Schema Registry, MirrorMaker2, Connect and directly in Kafka itself.
    In addition, i want to apply filters and sorting by type.

    2 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  2 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

11. Ability to choose Apache Kafka service user password length using Aiven Console

    As a developer,
    I want to be able to select password length and difficulty when creating a service user,
    so that I can stay complaint with internal security and compliance rules.

    2 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

12. Send a user alert in case stale credentials are being detected in connector configuration for Aiven for Apache Kafka Connect.

    As a developer,
    I want to be notified when I have stale credentials in my connectors,
    so that I can update them and make sure connection from a connector to a source or sink system could be successfully established.

    2 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

13. Debezium Connector for Cassandra

    As a developer,
    I want to connect to my Cassandra databases,
    so that I can use Change Data Capture (CDC) pattern to be notified about data changes and can ingest that data in to my data streaming pipeline for further use and processing.

    2 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

14. Monitoring consume lag for kafka not out of the box

    As a Solutions Architect
    I want to have consumer lag for kafka consumers available out of the box
    so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.

    Background
    1. there is a consumer lag on the default metrics dashboard but it does not work.
    2. I contact support and found that consumer lag is an early access feature that they enabled for me, but in order to get it into grafana it requires setting up an external prometheus(we cant do it all natively in Aiven)
    3. Here is support ticket for Reference: #89595

    As a Solutions Architect
    I want to have consumer lag for kafka consumers available out of the box
    so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.

    Background
    1. there is a consumer lag on the default metrics dashboard but it does not work.
    2. I contact support and found that… more

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  1 comment  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

15. Ability to scale Kafka cluster without upgrade

    As a Kafka Administrator
    I want to ahve ability to scalse Kafka clsuter without upgrade, that increses time of scale.
    In production this can become critical and lead to downtime. For example last 2 times the scale lasted for 12 hours.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

16. kafka_connect_connector_metrics availability over Prometheus

    Our customer Jago wants to monitor the status of connectors and tasks but currently can't find relevant metrics to do so. They want to be able to monitor the status of connectors and tasks on a dashboard and also get notified whenever a connector has not been running for X minutes.

    Jago has a connector running but can not find the metrics for kafka.connect:type=connector-metrics,connector=*.

    The specific metrics they are looking for is the one related to the status of a connector. For example, in the customers current self-managed kafka connect, they have the following metrics. This is convenient because they can get notified whenever a connector is not running.
    kafkaconnectconnector_metrics{status="running",connector="lfs-source-kafka-notifications",} 1.0

    They are also facing a similar issue with task status as they don't see metrics for it, while on their current cluster has:
    kafkaconnectconnector_status{connector="lfs-source-kafka-notification-created",status="running",task="0",} 1.0

    They have shared the attached metrics txt related to that connector that they got from prometheus.

    Our customer Jago wants to monitor the status of connectors and tasks but currently can't find relevant metrics to do so. They want to be able to monitor the status of connectors and tasks on a dashboard and also get notified whenever a connector has not been running for X minutes.

    Jago has a connector running but can not find the metrics for kafka.connect:type=connector-metrics,connector=*.

    The specific metrics they are looking for is the one related to the status of a connector. For example, in the customers current self-managed kafka connect, they have the following metrics. This is convenient because they… more

    • 

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  1 comment  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

17. Aiven's S3 sink connector - Configure `offset.flush.interval.ms` on the connector's level

    As a developer that uses Aiven's S3 sink connector,
    I want to be able to set the offset.flush.interval.ms only for my specific connector from the connector's configuration
    so that I can avoid configuring it in the cluster level (for all connectors).

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

18. Event log should show when a connector is paused and resumed.

    As a developer
    I want to know when a connector is paused or resumed
    so that I can have timestamps and know if anybody is doing what they are not supposed to do.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

19. Decompression transform for Connectors

    As an application developer
    I want to compress my kafka messages, but be able to decomrpess them using a transform before sinking them into a destination
    so that I can save on storage costs
    In addition, I'd like to use ZTSD, but more common libraries might be enough.

    Note, Confluent has something similar :
    https://docs.confluent.io/platform/current/connect/transforms/gzipdecompress.html

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

20. Automated dynamic quota configuration

    As a Cloud platform engineer, I need to have an automated way to set up and update quota configurations on a cluster, taking into account changes in resources consumption patterns amongst producers and consumers.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close