#dbt #cicd
Tldr

Atlassian Bamboo 에 구성한 dbt pipeline 을 Github Actions 으로 마이그레이션한 과정을 소개한다.

diagrams

|500

|500

Github Actions

shallow dive

변경 혹은 추가된 모델만 build 하는 slim-build Action 아래와 같고 라인별로 내용을 살펴본다.

아래 테스트한 내용은 추후 변경될 수 있음

name: dbt Slim ci
on:
  push:
    branches:
      - dev
      - main
    paths:
      - models/**
      - seeds/**
      - macros/**
      - docs/**
      - snapshots/**
jobs:
  build_model:
    runs-on: [self-hosted, linux, build]
    steps:
      - name: Check out repository code
        uses: actions/checkout@v3

      - name: Configure GCP credential for develop environment
        if: ${{ github.ref == 'refs/heads/dev' }}
        run: |
          echo "${{ secrets.DEV_DBT_KEYFILE }}" | base64 -d > ${{ vars.KEYFILE_PATH }}
          sudo mv ${{ vars.KEYFILE_PATH }} /var/run/secrets

      - name: Configure GCP credential for production environment
        if: ${{ github.ref == 'refs/heads/main' }}
        run: |
          echo "${{ secrets.PRD_DBT_KEYFILE }}" | base64 -d > ${{ vars.KEYFILE_PATH }}
          sudo mv ${{ vars.KEYFILE_PATH }} /var/run/secrets

      - name: Configure AWS credentials
        uses: actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ECR }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_KEY_ECR }}
          aws-region: ap-northeast-2

      - name: Login to ECR
        id: login-ecr
        uses: actions/amazon-ecr-login@v1

      - name: Set env to run container
        id: set-envs
        run: |
          echo "latest-author=$(git log -1 --pretty=format:%ae)" >> "$GITHUB_ENV"
          echo "repo-name=$(basename -s .git ${{ github.repository }})" >> "$GITHUB_ENV"

      - name: dbt slim ci
        id: build-image
        env:
          ECR_REGISTRY: ${{steps.login-ecr.outputs.registry}}
        run: |
          docker run --name dbt_slim_build \
            -v /var/run/secrets/keyfile:/app/keyfile \
            -v ${{ github.workspace	}}:/app \
            -v ${AWS_WEB_IDENTITY_TOKEN_FILE}:${AWS_WEB_IDENTITY_TOKEN_FILE} \
            -e AWS_WEB_IDENTITY_TOKEN_FILE=${AWS_WEB_IDENTITY_TOKEN_FILE} \
            -e AWS_ROLE_ARN=${AWS_ROLE_ARN} \
            -e DBT_TARGET=${GITHUB_REF_NAME} \
            -e JIRA_USERNAME=jj.lee \
            -e JIRA_TOKEN=${{ secrets.JIRA_TOKEN }} \
            -e JIRA_ASSIGNEE=${{ env.latest-author }} \
            ${{ env.ECR_REGISTRY }}/${{ env.repo-name }}:latest slim

name

on

jobs

steps

그 외 추가로 생성한 워크플로우

reusable workflow

Reusing workflows

PR auto labeler

container job

Running jobs in a container

SQL linter

Todo

docker container 형태로 실행되는 구조라 이미지를 빌드함. 워크플로우 실행시간이 길어 javascript 로 변환해서 사용하면 효율적

팁

reference