AIRFLOW
As Senior Data Engineer
I want to be able to orchestrate data flows (DAGS)
so that I can remove the dependency on dbt cloud and other separate tools that we currently run
In addition, dbt cloud is putting the price up making it prohibitory expensive and therefore we need a way to run dbt core and other things to make our data flow.
Philip Carrington
shared this idea
-
Alessandro Bacchini
commented
Aiven provides many services for processing and storing data, but it is missing a way for orchestrating them. I think that Airflow may be a good candidate as an orchestration tool.
The deployment of Airflow may be a little complex because it needs a web-server, a scheduler, a triggerer, celery, some workers and a database. However, for the initial implementation a simplified deployment may be considered, using the LocalExecutor. In this case, Airflow will be useful to schedule and to orchestrate processing jobs executed on other platforms like Flink, ClickHouse, MySQL, ...