Skip to content
← Aiven Ideas

Event Streaming

  • Aiven for Apache Kafka® - Apache Kafka as a fully managed service, deployed in the cloud of your choice and a full set of capabilities to build your streaming data pipelines. Find more info in our documentation or developer center.
  • Aiven for Apache Kafka® Connect - Seamlessly transport data with Kafka, integrate with external systems using Kafka Connect
  • Aiven for Apache Kafka® MirrorMaker 2 - replicate data with MirrorMaker 2
  • Karapace - easily manage schemas with Karapace
Join our forum to discuss your ideas with Aiven community or check out our public roadmap.

Event Streaming

Categories

JUMP TO ANOTHER FORUM

(thinking…)

I suggest you ...

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback

21 results found

  1. Support Databricks driver for Apache Kafka Connect JDBC sink connector

    As a developer,
    I want to have support for Databricks driver for Apache Kafka Connect JDBC sink connector,
    so that I can write data from Aiven for Apache Kafka to Databricks Spark for further processing, analysis and consumption.

    11 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  2. AWS SQS Source and Sink connector

    As a developer,
    I want to read and write data in AWS SQS
    so that I can enable more architectural solutions and use SQS with other AWS services.

    11 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  3. Support for Apache Kafka Connect to be run standalone against a 3rd party Apache Kafka service

    As a developer,
    I want to run standalone Apache Kafka Connect cluster against 3rd party Apache Kafka service,
    so that I can benefit from managed Apache Kafka Connect service and read/write data from/to Apache Kafka service running outside Aiven.

    10 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  4. Support for Protobuf serialization of Events (and Keys) in Aiven Kafka Connnect

    As a data engineer
    I want Aiven Kafka Connect to offer the option to use the protocol buffer data format when serializíng the events that is sends to a Kafka broker.
    In my specific case, I need it to be possible in a Debezium connector for PostgreSQL.
    Additionally, it would be good to have as a user the option to define oneself the protobuf schema to use for serializing.

    7 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  5. Salesforce Kafka Connector

    As a developer,
    I want to connect my Apache Kafka to Salesforce
    so that I can read and write data from/to SFDC.

    6 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  6. Azure Cosmos DB Kafka Connector

    As a developer,
    I want to connect to Azure Cosmos DB from Apache Kafka,
    so that I can access my data stored in Cosmos DB and use it in my stream processing pipeline.

    6 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  7. Aiven's S3 sink connector - Support configure offset flush max size

    As Developer
    I want to use Aiven's S3 sink connector and have the ability to control the size of output files (keep same size even if the traffic changes). In general, the connector has no lags and we want to flush the offset and write to file only when we have enough data. We use offset.flush.interval.ms for it but when the traffic increases, the amount of data arrives in the configured interval is increased and can cause an OOM issue. In addition, when we pause the connector for couple of minutes and we gather a lag, it can also lead…

    5 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  8. Azure Data Lake Gen2 (ADLS) Kafka connector

    Kafka Connector to allow for streaming data to Azure Data Lake Gen2 as a sink.

    5 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  9. Apache Camel connector for Kafka Connect

    As a software / data science engineer,
    I want to be able to connect Apache Kafka to Apache Camel,
    so that I can define my rules, routing and ETL pipeline to process and route data from and to different systems.

    5 votes
    Gathering Interest  ·  1 comment  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  10. SAP Source connector for Aiven for Apache Kafka Connect

    As a developer,
    I want to be able to pull data from SAP systems,
    so I could build my data pipelines.

    4 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  11. Kafka Connect GCS Sink: Support using field values to define bucket name or file name prefix

    As data streaming architect
    I want to be able to export records from Kafka to GCS and use values in the record to define the bucket or file name
    so that I can organize data by those values to make them easier to find and process.

    Use case is a multi-user/multi-tenant application where user info is a value in the record. Need to be able to organize the output in object storage by that value somehow.

    3 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  12. Add OpenTelemetry API and SDK to enable traceability in Debezium connector

    As IT Architect
    I want to provide complete traceability within my microservice mesh where 2 microservice communicate via Outbox pattern via Debezium connector. In order to achieve this, Debezium connector requires some of the OpenTelemtry APIs are on the Kafka Connect classpath.
    I want to have this feature so that I can see complete chain of interactions for specific request, observe what and where time is spent inside particular microservice and find out possible bottlenecks.

    3 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  13. Aiven API support for Kafka Connect Java clients

    As an AIven developer user
    I want to use the Java code generated from Aiven API to communicate with an Aiven managed KafkaConnect cluster
    so that I can build micro services in Java platform to manage various connectors on the cluster
    In addition, this gives more security for Aiven users and data while the standard REST API supports only BasicAuth.

    2 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  14. Ease the setup of Kafka Connectors when using internal Aiven services

    As a developer,
    I want to have an easy setup of my Kafka connectors when they are using internal Aiven services such as Postgres or OpenSearch
    so that I can save time, avoid mistakes, and enforce the strength of the platform.

    2 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  15. Send a user alert in case stale credentials are being detected in connector configuration for Aiven for Apache Kafka Connect.

    As a developer,
    I want to be notified when I have stale credentials in my connectors,
    so that I can update them and make sure connection from a connector to a source or sink system could be successfully established.

    2 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  16. Debezium Connector for Cassandra

    As a developer,
    I want to connect to my Cassandra databases,
    so that I can use Change Data Capture (CDC) pattern to be notified about data changes and can ingest that data in to my data streaming pipeline for further use and processing.

    2 votes
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  17. Aiven's S3 sink connector - Configure `offset.flush.interval.ms` on the connector's level

    As a developer that uses Aiven's S3 sink connector,
    I want to be able to set the offset.flush.interval.ms only for my specific connector from the connector's configuration
    so that I can avoid configuring it in the cluster level (for all connectors).

    1 vote
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  18. Event log should show when a connector is paused and resumed.

    As a developer
    I want to know when a connector is paused or resumed
    so that I can have timestamps and know if anybody is doing what they are not supposed to do.

    1 vote
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  19. Decompression transform for Connectors

    As an application developer
    I want to compress my kafka messages, but be able to decomrpess them using a transform before sinking them into a destination
    so that I can save on storage costs
    In addition, I'd like to use ZTSD, but more common libraries might be enough.

    Note, Confluent has something similar :
    https://docs.confluent.io/platform/current/connect/transforms/gzipdecompress.html

    1 vote
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?

  20. io.debezium.transforms.partitions.PartitionRouting

    It would be great it the io.debezium.transforms.partitions.PartitionRouting SMT was available for use when configuring a Kafka Debezium source connector on the Aiven platform.

    1 vote
    Gathering Interest  ·  0 comments  ·  Aiven for Apache Kafka® Connect  ·  Admin →
    How important is this to you?
← Previous 1
  • Don't see your idea?