15 results found

1. Monitoring consume lag for kafka not out of the box

   As a Solutions Architect
   I want to have consumer lag for kafka consumers available out of the box
   so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.

   Background
   1. there is a consumer lag on the default metrics dashboard but it does not work.
   2. I contact support and found that consumer lag is an early access feature that they enabled for me, but in order to get it into grafana it requires setting up an external prometheus(we cant do it all natively in Aiven)
   3. Here is support ticket for Reference: #89595

   As a Solutions Architect
   I want to have consumer lag for kafka consumers available out of the box
   so that I can correctly monitor streaming applications without having to setup an external prometheus. In addition Consumer lag is the key metric to monitor for end to end health of streaming apps(ie to ensure they are keeping up with demand), you cannot put a streaming app into production with out correct monitoring and alerting on this metric.

   Background
   1. there is a consumer lag on the default metrics dashboard but it does not work.
   2. I contact support and found that… more

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  1 comment  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

2. Exactly-Once support in Storage Write API from our GBQ sink connector

   Add support for exactly-once delivery in Storage Write API for GBQ sink connector.

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  1 comment  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

3. Kafka tiered storage with external S3 bucket

   As a developer
   I want to have the ability to use our own S3 bucket for storing Kafka tiered storage
   so that I can access the data from S3 and query some data for debugging (without streaming all the data to Kafka)

   3 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

4. Support for both allow and black lists in Kafka service

   As a managed Kafka administrator
   I want to have the ability to black-list a bunch of IP addresses I believe are suspicious from my Kafka service, so that I can prevent my service being disrupted by unexpected traffic
   In addition, I would like to keep the current allow-list to be able to allow know IPs and a way to resolve a conflict between allow and block list where block list takes priority.

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close
   Shelved  ·  AdminMichael Tansini (Senior Product Manager, Aiven.io) responded

   at the moment we recommend using our current networking whitelisting capabilities

5. Add Datadog integration to Flink

   As a data engineer
   I want to use my integrate my existing Datadog subscription with Flink
   so that I can store and monitor all metrics across my stack in a single location.
   In addition, this functionality is already available on other services in Aiven

   3 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

6. ClickHouse Driver for Apache Kafka Connect JDBC Sink connector

   As a developer,
   I want to have ClickHouse driver support in JDBC sink connector,
   so that I can write data from Apache Kafka to ClickHouse for further processing and analysis.

   7 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  2 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

7. A no-code solution for Flink to unlock the usage for non-technical users

   As a non-technical user
   I want to be able to aggregate and join different streams of data
   without the need of developers

   1 vote

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   1 comment  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close
   Shelved  ·  Jonah Kowall responded

   This function will largely be replaced by ChatGPT or other LLMs which can generate clear code and instructions, making a visual builder unnecessary. 

8. Add Oracle JDBC support for Aiven for Kafka Connect

   As a developer,
   I want to be able to connect to my Oracle database from Aiven for Apache Kafka Connect,
   so that I can read and write data from/to Oracle DB and/or Apache Kafka to enable my use-cases.

   9 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

9. Customer is interested to have Pyflink be supported with Flink

   As a Developer I want to use Pyflink library with Aiven for Apache Flink so that I can directly use it in my project.

   7 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   Shelved  ·  1 comment  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
   How important is this to you?

   Nice to have You must login first! Should have You must login first! Must have You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

10. Flink HTTP API sink

    As a developer I would like to push the output of a Flink operation to an HTTP API sink.

    3 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Shelved  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

11. Field Level Encryption Support for Aiven Products

    TL;DR;
    As Aiven Customer
    I want to be able to be able to encrypt any form of sensitive data (PII or PCI) so that I can manage sensitive data in a legally compliant and user-privacy respectful manner.

    Detailed description of the proposal:
    Hi Aiven!
    Hope all is well with you. I have a feature/service suggestion which I believe will make your existing product portfolio even stronger!

    It’s a thing I call “Aiven for Privacy FTW!” and it’s basically a standalone “field-level-encryption” service used for managing of PII and PCI fields/properties in the event payload in legally (eg. GDPR and/or CCPA) and secure manner (ie. sensitive data encrypted in transit and at rest).

    I made a following high-level architecture/component diagram with aim to visualize how would “Aiven for Privacy FTW!” fit together with your products and within client context. See attached.

    After drawing and discussing this internally, I realized that it would be even better if this is a stand-alone service that can be instantiated and run fully within the client context and manage sensitive fields/properties even before event enters Aiven managed services. This way, the sensitivity will be fully managed and stay within the client/producer control and will not pollute your services with sensitive data.

    Why should you consider this? I’d like to claim that you’re in unique position in the industry with a portfolio of solutions that can be connected fully end-to-end from event producer to consumer and with this type of functionality you’ll be able to provide a turn-key solution to all your clients.

    Nothing here is new, tbh. There is no novelty here and others are doing/offering something similar already, but no one is giving a full package solution with privacy compliance being made a first class citizen. You could.

    Example of others doing stuff in this area:
    - Snowflake allows their customers to use external tokenizers to unlock/show sensitive data based on the access privilege: https://docs.snowflake.com/en/user-guide/security-column-ext-token-intro
    - MongoDB is offering their clients ability to encrypt sensitive data on the client side using: https://www.mongodb.com/docs/manual/core/csfle/
    - Conduktor provides encryption all the way down to field/property level: https://www.conduktor.io/encryption/

    In all these cases (at least MongoDB and Snowflake), “Aiven for Privacy FTW!” if it becomes a thing could be used if done as a stand-alone service. So, it could strengthen your position further in the industry.

    I hope you will consider this and feel free to contact me either directly or through our Aiven KAM, Hanna Behr, if you want to talk more about this.

    Thanks,
    Srdjan B. (Eng. Director, Sinch SaaS App Platform)

    TL;DR;
    As Aiven Customer
    I want to be able to be able to encrypt any form of sensitive data (PII or PCI) so that I can manage sensitive data in a legally compliant and user-privacy respectful manner.

    Detailed description of the proposal:
    Hi Aiven!
    Hope all is well with you. I have a feature/service suggestion which I believe will make your existing product portfolio even stronger!

    It’s a thing I call “Aiven for Privacy FTW!” and it’s basically a standalone “field-level-encryption” service used for managing of PII and PCI fields/properties in the event payload in legally (eg. GDPR and/or CCPA)… more

    •  

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    1 comment  ·  Other  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close
    Shelved  ·  Jonah Kowall responded

    I will be closing this out, but the idea is valid. We are looking at building a proxy service for Kafka and this could be in part of the roadmap for that component as you have described. We suggest for those who want to use encryption on Kafka to do this on the producer and consumer sides as the data would be encrypted from end to end. 

12. Support S3 as a source and sink for Flink

    As a developer I want to be able to read and write data to my S3 object storage, in order to simply integrate Flink into my existing data architecture. Using Flink to read data from S3, transform it, and then write to another S3 location allows easy consolidation and data quality management in a common reference data architecture.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Shelved  ·  0 comments  ·  Aiven for Apache Flink®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

13. Sending Apache Kafka metrics to Datadog

    As Software Engineer at Wex
    I want to send additional metrics to Datadog
    so that I can send important metrics from Aiven's cluster as these ones:
    IsrShrinksPerSec
    IsrExpandsPerSec
    ActiveControllerCount
    OfflinePartitionsCount
    TotalTimeMs
    PurgatorySize
    RequestsPerSec
    Network bytes sent/received
    BytesInPerSec/BytesOutPerSec

    According to Datadog's documentation, these metrics are considered highly significant.
    https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/#kafka-emitted-metrics

    2 votes

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Shelved  ·  0 comments  ·  Aiven for Apache Kafka®  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

14. Rockset Sink connector for Kafka Connect

    As a data engineer,
    I want to write data to Rockset,
    so that I can run my analytical workloads.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Shelved  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

15. Apache NiFi Source and Sink Kafka connector

    As a developer
    I want to use Apache NiFi Kafka Connector
    so that I can move data in and out Apache NiFi to orchestrate data flow.

    1 vote

    Vote Vote Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close

    Vote

    We’ll send you updates on this idea

    Shelved  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Delete…  ·  Admin →
    How important is this to you?

    Nice to have You must login first! Should have You must login first! Must have You must login first!

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    Close

    Close