Github actions for dbt sync

This is a step-by-step guideline for using Github Actions to synchronise that file with GCS and always have a data catalog up-to-date.

Create the action itself

You need to create a folder in your dbt repository with the path <repository>/.github/workflows/

Inside you need to create a YAML file, we will name ours send_to_catalog.yml

The content is :

name: Send manifest to Catalog
on: [push]
jobs:
  send-manifest-to-catalog:
    runs-on: ubuntu-latest
    steps:
      - name: Imports Github code source
        uses: actions/checkout@v2 
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install castor-extractor
      - name: Launch python script
        run: castor-upload -c '${{ secrets.SERVICE_ACCOUNT_JSON }}' -s <source_id> -f dbt_manifest.json -t DBT

Create the secrets

There's a reference guide in order to understand the manipulation of secrets using Github Actions.

Here's a screenshot of the minified version of the Service Account JSON content stored and passed to the Python function.

Push the code

Once that all the code is pushed to the repository, the workflow will be triggered. Then you will see something like this in the Actions tab of the repository.

The date and time of the file will be in UTC