Handling YAML on Linux: Why yq is the Perfect Replacement for sed and awk? – ITFROMZERO

Table of Contents

Why You Should Stop Using sed or awk to Edit YAML Files

If you work with Kubernetes, Docker Compose, or CI/CD pipelines, YAML is unavoidable. Previously, I often used sed or awk for quick value updates. However, YAML is extremely sensitive to whitespace and indentation. A single sed command that replaces the wrong part or shifts a space can bring the entire system down immediately.

I once encountered an incident on an Ubuntu 22.04 production server. At the time, I used sed to update an image version in a Deployment file. Because it didn’t understand the nested structure, sed ended up changing similar character strings in unrelated blocks. As a result, the system experienced 15 minutes of downtime just because of a formatting error. That’s why you need a tool that understands the tree structure of YAML instead of treating it as plain text. yq is the most precise solution.

Comparing Common Ways to Handle YAML Files

1. Manual Editing (Vim/Nano)

Pros: Intuitive for short files.
Cons: Cannot be automated. When you need to edit 100 files at once, this method is completely impossible.

2. Using sed/awk/grep

Pros: Always available on Linux.
Cons: Easily breaks YAML structure. Writing Regex for nested levels is a nightmare and highly prone to errors.

3. Using yq

Pros: Query and modify based on logical structure. Syntax is similar to jq, ensuring absolute file format protection.
Cons: Requires installation, though the binary is only about 10-15MB.

Choosing the Right yq Version

There are currently two versions of yq: one written in Python and one written in Go (by Mike Farah). In this article, I am using Mike Farah’s version. This is the standard version in the DevOps community because it runs independently, requires no additional runtime, and has extremely fast processing speeds.

Install yq in 30 Seconds

Instead of using outdated OS repositories, I always prefer downloading the binary directly from GitHub to get the latest features:

# Download the binary for x86_64 architecture
sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq

# Grant execution permissions
sudo chmod +x /usr/bin/yq

# Verify installation
yq --version

Practical Tasks with yq

Let’s use the following config.yaml file as a practical example:

app_name: my-web-app
environment: staging
replica_count: 2
services:
  - name: frontend
    port: 80
  - name: backend
    port: 8080
database:
  host: localhost
  user: admin

1. Reading Data Accurately

You can retrieve the value of any field using the dot (.) operator. This is extremely useful when you need to assign values to variables in shell scripts.

# Get application name
yq '.app_name' config.yaml

# Get port of the first service (index 0)
yq '.services[0].port' config.yaml

2. Updating Values Without Breaking Format

This is the most critical task. For example, you want to increase replica_count to 5 and change the environment to “production”.

# Preview the changes on screen
yq '.replica_count = 5' config.yaml

# Apply changes directly to the file using the -i flag
yq -i '.environment = "production"' config.yaml

3. Adding and Deleting Fields Quickly

Add a database timeout configuration or remove sensitive information with a single command:

# Add a new field
yq -i '.database.timeout = 30' config.yaml

# Delete the user field for security
yq -i 'del(.database.user)' config.yaml

4. Converting YAML to JSON

Many APIs only accept JSON, but your configuration files are in YAML. yq handles this in an instant.

# Export in JSON format
yq -o=json config.yaml

# Convert back from JSON to pretty-printed YAML
yq -P config.json

Tips from Real-World Experience

After years of infrastructure management, I’ve gathered a few tips to help you use yq more safely:

Dry-run before overwriting: Always run the command without the -i flag to check the output. Once you’re sure the logic is correct, then apply the overwrite to the original file.
Leverage Pipes (|): yq works very well with other Linux commands. You can use curl to fetch a remote configuration file and pipe it directly into yq to filter data.
Merge multiple configuration files: yq supports merging environment files (base and override) very effectively.

Example of merging general and environment-specific configurations:

yq eval-all '. as $item ireduce ({}; . * $item)' base.yaml override.yaml

Conclusion

Using yq makes a SysAdmin’s job more professional and significantly reduces risk. Instead of struggling with complex sed lines, let yq handle data according to its structural nature. If you are working with Infrastructure as Code (IaC), this is definitely a tool you should have in your toolkit.