End ‘YAML Hell’ with Dagger: Building CI/CD Pipelines using Python/Go – ITFROMZERO

Table of Contents

The Problem: The “Works on my machine, but fails in CI” Nightmare

Almost every developer has experienced this feeling. You push code to GitHub, wait 5 minutes for CI to run, only to get a cold splash of water from a minor error. You fix a line of YAML, push again, and pray. This cycle can repeat up to 20 times just to configure a simple script.

The core issue lies in the tools. GitHub Actions, GitLab CI, or Jenkins all rely on YAML — a data format, not a programming language. As systems grow, YAML files become a “tangled mess” that is extremely difficult to debug. More dangerously, it’s nearly impossible to run that exact pipeline locally. Everything depends entirely on the provider’s Cloud infrastructure.

Why does YAML-based CI/CD cause so much fatigue?

Lack of abstraction: YAML has no functions or types. Handling complex branching logic in YAML feels like writing code with your non-dominant hand.
Environment drift: Cloud runners often use different OS versions and libraries than local machines. As a result, code runs smoothly on your machine but crashes upon deployment.
Slow feedback: Every small change requires a commit and push. You waste hours just waiting for runners to start.

Dagger: Returning CI/CD to Its True Nature as Source Code

Dagger completely changes the game. Instead of writing YAML, you write code in the languages you’re already proficient in, such as Python, Go, or TypeScript. The Dagger engine uses Docker (via BuildKit) to execute build steps. This ensures that whether you run it locally or on Jenkins, the results are always 100% consistent.

In a microservices project with over 30 containers I once managed, switching to Dagger brought immediate results. Average build time dropped from 15 minutes to under 5 minutes. BuildKit’s smart caching mechanism helped us save up to 40% in system resources.

Installing Dagger on Your Local Machine

First, ensure you have Docker installed. Then, install the Dagger CLI to coordinate code execution.

# Install for Linux/macOS
curl -L https://dl.dagger.io/dagger/install.sh | sh
sudo mv bin/dagger /usr/local/bin/dagger

# Verify successful installation
dagger version

Initialize Dagger directly in your project directory using the command:

dagger init

Configuring the Pipeline with the Python SDK

Let’s see how Dagger replaces a traditional .github/workflows/ci.yml file. Instead of declaring YAML, we create a ci/main.py file to manage the steps: Install -> Unit Test -> Build Image.

import sys
import anyio
import dagger

async def main():
    # Initialize Dagger client
    async with dagger.connection(dagger.Config(log_output=sys.stderr)) as client:
        
        # Get root directory
        src = client.host().directory(".")

        # Define Python 3.11 environment
        python_base = (
            client.container()
            .from_("python:3.11-slim")
            .with_directory("/app", src)
            .with_workdir("/app")
        )

        # Step 1: Install dependencies
        runner = (
            python_base
            .with_exec(["pip", "install", "-r", "requirements.txt"])
        )

        # Step 2: Run Unit Tests
        print("Running tests...")
        test_result = await runner.with_exec(["pytest", "tests/"]).stdout()
        print(test_result)

        # Step 3: Build final image
        image = runner.with_entrypoint(["python", "main.py"])

if __name__ == "__main__":
    anyio.run(main)

The Power Lies in the Caching Mechanism

Dagger handles each with_exec command in ephemeral containers. If requirements.txt hasn’t changed, Dagger reuses the cache instantly. In subsequent runs, heavy library installation steps will take only seconds instead of minutes.

Local Debugging: No More Guesswork

This is the most valuable feature. To test the pipeline, you just need to type a terminal command on your machine:

python ci/main.py

All logs are displayed directly. If a test step fails, you fix the code and rerun it immediately without needing a git push. Dagger also supports interactive mode. When a step fails, you can jump straight into that container to inspect the file system, just like using docker exec.

# Access the failing environment directly for inspection
dagger call test --interactive

Deploying to the Cloud (GitHub Actions/GitLab CI)

Once the pipeline runs stably locally, moving it to the Cloud becomes incredibly smooth. At this point, GitHub Actions simply acts as a basic runner. The business logic is encapsulated within the Python code.

name: CI
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Dagger Pipeline
        uses: dagger/dagger-for-github@v5
        with:
          verb: run
          args: python ci/main.py

This approach frees you from vendor lock-in. If your team wants to switch to GitLab CI later, you just need to copy the Python file over. No need to relearn YAML syntax or modify logic.

Key Takeaways from Practical Experience

After over a year of implementing Dagger in production, I’ve distilled three golden rules:

Break it down for caching: Don’t combine too many commands into a single with_exec. The more you break it down, the more effectively Dagger can reuse the cache.
Manage Secrets securely: Never hardcode passwords. Utilize the client.set_secret() function to secure sensitive information.
Local First: Always ensure the pipeline passes on your local machine before pushing. This habit keeps the commit history clean and saves time for the whole team.

Dagger is not just a new tool. It represents the mindset: CI/CD is software. When you treat a pipeline as software, you gain the power to use unit tests, linters, and debuggers to control it. Free yourself from dry YAML files today.