85 results found

1. Sending Apache Kafka metrics to Datadog

   As Software Engineer at Wex
   I want to send additional metrics to Datadog
   so that I can send important metrics from Aiven's cluster as these ones:
   IsrShrinksPerSec
   IsrExpandsPerSec
   ActiveControllerCount
   OfflinePartitionsCount
   TotalTimeMs
   PurgatorySize
   RequestsPerSec
   Network bytes sent/received
   BytesInPerSec/BytesOutPerSec

   According to Datadog's documentation, these metrics are considered highly significant.
   https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/#kafka-emitted-metrics

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

2. Ability to choose Apache Kafka service user password length using Aiven Console

   As a developer,
   I want to be able to select password length and difficulty when creating a service user,
   so that I can stay complaint with internal security and compliance rules.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

3. Send a user alert in case stale credentials are being detected in connector configuration for Aiven for Apache Kafka Connect.

   As a developer,
   I want to be notified when I have stale credentials in my connectors,
   so that I can update them and make sure connection from a connector to a source or sink system could be successfully established.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

4. Debezium Connector for Cassandra

   As a developer,
   I want to connect to my Cassandra databases,
   so that I can use Change Data Capture (CDC) pattern to be notified about data changes and can ingest that data in to my data streaming pipeline for further use and processing.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

5. Support for "Apache Iceberg" format while sinking CDC

   As a developer / DevOps
   I want to be able to Sink CDC data into Apache Iceberg format
   so that I can analyze data using time travel feature of AWS Athena
   In addition, we may find a way for the current "Aiven - Amazon AWS S3 Sink" connector to be able to produce "Apache Iceberg" in addition of "Parquet" format or we may provide a dedicated connector like the one from this repository : https://github.com/tabular-io/iceberg-kafka-connect

   Yours faithfully,
   LCDP

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

6. Support for both allow and black lists in Kafka service

   As a managed Kafka administrator
   I want to have the ability to black-list a bunch of IP addresses I believe are suspicious from my Kafka service, so that I can prevent my service being disrupted by unexpected traffic
   In addition, I would like to keep the current allow-list to be able to allow know IPs and a way to resolve a conflict between allow and block list where block list takes priority.

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Aiven for Apache Kafka®  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

7. Cluster leader balancing CPU vs Disk - can we choose?

   As an OPS Engineer
   I want to be able to balance the cluster based on CPU usage rather than Disk usage. The current algorithm focuses on Disk usage which is not optimum for our application.
   Can we have an option to apply partition rebalancing based on CPU usage?

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Aiven for Apache Kafka®  ·  Edit…  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

8. Ability to scale Kafka cluster without upgrade

   As a Kafka Administrator
   I want to ahve ability to scalse Kafka clsuter without upgrade, that increses time of scale.
   In production this can become critical and lead to downtime. For example last 2 times the scale lasted for 12 hours.

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

9. Support for RocksDB in Flink

   Support RocksDB as a persistent data store for Flink

   As an engineer I need be able to run larger state Flink jobs in order to meet my data processing requirements

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Gathering Interest  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

10. kafka_connect_connector_metrics availability over Prometheus

    Our customer Jago wants to monitor the status of connectors and tasks but currently can't find relevant metrics to do so. They want to be able to monitor the status of connectors and tasks on a dashboard and also get notified whenever a connector has not been running for X minutes.

    Jago has a connector running but can not find the metrics for kafka.connect:type=connector-metrics,connector=*.

    The specific metrics they are looking for is the one related to the status of a connector. For example, in the customers current self-managed kafka connect, they have the following metrics. This is convenient because they can get notified whenever a connector is not running.
    kafkaconnectconnector_metrics{status="running",connector="lfs-source-kafka-notifications",} 1.0

    They are also facing a similar issue with task status as they don't see metrics for it, while on their current cluster has:
    kafkaconnectconnector_status{connector="lfs-source-kafka-notification-created",status="running",task="0",} 1.0

    They have shared the attached metrics txt related to that connector that they got from prometheus.

    Our customer Jago wants to monitor the status of connectors and tasks but currently can't find relevant metrics to do so. They want to be able to monitor the status of connectors and tasks on a dashboard and also get notified whenever a connector has not been running for X minutes.

    Jago has a connector running but can not find the metrics for kafka.connect:type=connector-metrics,connector=*.

    The specific metrics they are looking for is the one related to the status of a connector. For example, in the customers current self-managed kafka connect, they have the following metrics. This is convenient because they… more

    • 

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  1 comment  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

11. Aiven's S3 sink connector - Configure `offset.flush.interval.ms` on the connector's level

    As a developer that uses Aiven's S3 sink connector,
    I want to be able to set the offset.flush.interval.ms only for my specific connector from the connector's configuration
    so that I can avoid configuring it in the cluster level (for all connectors).

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

12. A no-code solution for Flink to unlock the usage for non-technical users

    As a non-technical user
    I want to be able to aggregate and join different streams of data
    without the need of developers

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    1 comment  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close
    Shelved  ·  Jonah Kowall responded

    This function will largely be replaced by ChatGPT or other LLMs which can generate clear code and instructions, making a visual builder unnecessary. 

13. Event log should show when a connector is paused and resumed.

    As a developer
    I want to know when a connector is paused or resumed
    so that I can have timestamps and know if anybody is doing what they are not supposed to do.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

14. Decompression transform for Connectors

    As an application developer
    I want to compress my kafka messages, but be able to decomrpess them using a transform before sinking them into a destination
    so that I can save on storage costs
    In addition, I'd like to use ZTSD, but more common libraries might be enough.

    Note, Confluent has something similar :
    https://docs.confluent.io/platform/current/connect/transforms/gzipdecompress.html

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

15. Add external Delta Lake connector for Apache Flink

    As a developer, I need to be able to read and write from Databricks so that I can complement my existing data architecture

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

16. Auto-Scaling

    As a developer, I need Flink to be able to automatically increase compute resources to a self-defined threshold to seamlessly process significant increases in traffic in order to maintain performance. This increase in traffic would be for either a permanently higher workload, or for a temporary pre-defined amount of time, and would be triggered by a customer request, with subsequent additional billing impacts

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

17. Automated dynamic quota configuration

    As a Cloud platform engineer, I need to have an automated way to set up and update quota configurations on a cluster, taking into account changes in resources consumption patterns amongst producers and consumers.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

18. Visualization of client IDs that interact with a Kafka cluster

    As a Cloud platform engineer, I need to have a view of Kafka cluster resources consumption across all clients IDs. Such information is needed to configure quotas and to manage them.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

19. Field Level Encryption Support for Aiven Products

    TL;DR;
    As Aiven Customer
    I want to be able to be able to encrypt any form of sensitive data (PII or PCI) so that I can manage sensitive data in a legally compliant and user-privacy respectful manner.

    Detailed description of the proposal:
    Hi Aiven!
    Hope all is well with you. I have a feature/service suggestion which I believe will make your existing product portfolio even stronger!

    It’s a thing I call “Aiven for Privacy FTW!” and it’s basically a standalone “field-level-encryption” service used for managing of PII and PCI fields/properties in the event payload in legally (eg. GDPR and/or CCPA) and secure manner (ie. sensitive data encrypted in transit and at rest).

    I made a following high-level architecture/component diagram with aim to visualize how would “Aiven for Privacy FTW!” fit together with your products and within client context. See attached.

    After drawing and discussing this internally, I realized that it would be even better if this is a stand-alone service that can be instantiated and run fully within the client context and manage sensitive fields/properties even before event enters Aiven managed services. This way, the sensitivity will be fully managed and stay within the client/producer control and will not pollute your services with sensitive data.

    Why should you consider this? I’d like to claim that you’re in unique position in the industry with a portfolio of solutions that can be connected fully end-to-end from event producer to consumer and with this type of functionality you’ll be able to provide a turn-key solution to all your clients.

    Nothing here is new, tbh. There is no novelty here and others are doing/offering something similar already, but no one is giving a full package solution with privacy compliance being made a first class citizen. You could.

    Example of others doing stuff in this area:
    - Snowflake allows their customers to use external tokenizers to unlock/show sensitive data based on the access privilege: https://docs.snowflake.com/en/user-guide/security-column-ext-token-intro
    - MongoDB is offering their clients ability to encrypt sensitive data on the client side using: https://www.mongodb.com/docs/manual/core/csfle/
    - Conduktor provides encryption all the way down to field/property level: https://www.conduktor.io/encryption/

    In all these cases (at least MongoDB and Snowflake), “Aiven for Privacy FTW!” if it becomes a thing could be used if done as a stand-alone service. So, it could strengthen your position further in the industry.

    I hope you will consider this and feel free to contact me either directly or through our Aiven KAM, Hanna Behr, if you want to talk more about this.

    Thanks,
    Srdjan B. (Eng. Director, Sinch SaaS App Platform)

    TL;DR;
    As Aiven Customer
    I want to be able to be able to encrypt any form of sensitive data (PII or PCI) so that I can manage sensitive data in a legally compliant and user-privacy respectful manner.

    Detailed description of the proposal:
    Hi Aiven!
    Hope all is well with you. I have a feature/service suggestion which I believe will make your existing product portfolio even stronger!

    It’s a thing I call “Aiven for Privacy FTW!” and it’s basically a standalone “field-level-encryption” service used for managing of PII and PCI fields/properties in the event payload in legally (eg. GDPR and/or CCPA)… more

    •  

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    1 comment  ·  Other  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close
    Shelved  ·  Jonah Kowall responded

    I will be closing this out, but the idea is valid. We are looking at building a proxy service for Kafka and this could be in part of the roadmap for that component as you have described. We suggest for those who want to use encryption on Kafka to do this on the producer and consumer sides as the data would be encrypted from end to end. 

20. io.debezium.transforms.partitions.PartitionRouting

    It would be great it the io.debezium.transforms.partitions.PartitionRouting SMT was available for use when configuring a Kafka Debezium source connector on the Aiven platform.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close