#AIRFLOW HELM CHART VALUES HOW TO#

This stage is for those who deployed the Gitlab bot on Kubernetes.Īs we use Amazon Route 53 for a DNS service, we are utilising the following ClusterIssuer to issue a TLS certificate for the given domain.

(3) Setting up an Ingress and ClusterIssuer to redirect our sub-domain traffic to the above server and automatically issue a TLS certificate with Let’s Encrypt Akrossy responding to an issue “open” event Now we can successfully test the webhook by simply opening a new issue in the same repository like below. (Temporary) Webhook registered for Airflow DAG repository The above example only includes the implementation of an Issue Hook (Open) event for now and to test, we can use something like ngrok as a temporary reverse proxy for fronting this skeleton server.īefore testing, please make sure you have registered a webhook at the Airflow DAG repository in Project - Settings - Webhooks. GL_ACCESS_TOKEN = os.getenv('GL_ACCESS_TOKEN') GITLAB_PROJECT_ID = XXXXXXXX # Edit to your project ID (2) Writing a skeleton for a custom Gitlab Bot using gidgetlabīefore stepping in, the documentation and guide for gidgetlab can be found here in both its documentation and blog.ĪKROSSY_USER_ID = XXXXXXXX # Edit to your bot user ID User Settings - SSH KeysĪlso, don’t forget to create a new Access Token with API scope in User Settings - Access Tokens - Personal Access Tokens User Settings - Access Tokens - Personal Access Tokens Then we need to generate an SSH key pair and register it on the above account in User Settings - SSH Keys. We first need a new Gitlab account to be used as a bot, so simply go to and create one! Introducing our Gitlab bot Akrossy!

Using a custom implementation of the gidgetlab using Starlette.

Triggering Gitlab pipelines using MR-related events (Deployment & Withdrawal).

Scripts to be used by Gitlab Runner for deploying and withdrawing a test Airflow environment.

Preparing an override values YAML for an Airflow Helm Chart.

Registering a webhook on the (Airflow DAGs) Gitlab repository.

Setting up an Ingress and ClusterIssuer to redirect subdomain traffic to the above server and automatically issue a TLS certificate with Let’s Encrypt.

Writing a skeleton for a custom Gitlab Bot using gidgetlab.

In this post, I will describe the whole process briefly in the following order. Not knowing how this could be done with Gitlab, followed by some research, we were successfully able to launch Akros’ first Gitlab Bot, Akrossy with an architecture described in the above diagram! Inspired by a tech blog post from NFTBank, it seemed fascinating to be able to deploy and withdraw test environments within a merge request ( pull request).

Overview A simplified diagram describing the high-level architecture Thus, we have decided to provide easy access to a temporary Airflow while automating the process with Gitlab CI and a custom Gitlab Bot API (On top of the gidgetlab written by Benjamin Bertrand). Witnessing the above issues reducing each others’ productivity, we thought an isolated test environment for Airflow is required for everyone to safely develop and test Airflow DAGs. This made engineers very cautious before testing a new pipeline on a single repository. One problem was that making changes to an already running Airflow DAG could affect this DAG in a way that either a scheduler changing the task status to removed or new tasks being executed within the already running DAG.Īs git-sync is enabled (Git integration to synchronize DAGs), it was becoming troublesome for multiple engineers to commit and push their changes without affecting each other. When we first deployed Apache Airflow on our RKE2 Kubernetes cluster, everything seemed flawless in the beginning, however, soon later we started to face problems, especially when multiple engineers were developing and testing Airflow DAGs on the same repository. In Akros Technologies, every engineer uses Apache Airflow as a workflow scheduler for various tasks such as running alpha-inference, portfolio operation, and building financial data pipelines.

#AIRFLOW HELM CHART VALUES HOW TO#

How to Deploy Automated & Isolated Airflow Test Environments with Gitlab Bot on Kubernetes

Airflow helm chart values