Support for 3rd party ETL tools with Aiven for ClickHouse
As an Data Engineer
I want to use 3rd party ETL tools to populate Clickhouse tables
so that I can use clickhouse as my warehouse. Currently popular tools such as airbyte do not work when connected to Aiven clickhouse due to the fact that it creates internal state tables within clickhouse, and it fails given that tables are not allow to be created outside the context of the Aiven console. Customers are unable to use external tools that can self manage state and required the permission.
2024-01-26 22:37:45 normalization > Code: 497. DB::Exception: avnadmin: Not enough privileges. To execute this query, it's necessary to have the grant CREATE DATABASE ON airbytedefault.*. (ACCESS_DENIED) (version 23.8.8.1)
AdminJason Hepp
(Senior Director of Solution Architecture, Aiven.io)
shared this idea
-
Paul Rogalinski-Pinter
commented
The permissions to create a table are available to 3rd party tools. We have have validated Meltano and DBT to work with Aiven for ClickHouse. Our initial Airbyte evaluation for simple data movement tasks did also work just fine.
The specific error above is related to assigning `CREATE DATABASE` permissions and is a bit confusing TBH.