Toil Elimination: Automating the Work Nobody Should Be Doing

Toil elimination is the practice of identifying and automating repetitive, manual, and low-value work that teams keep doing but should not have to. It matters when engineers spend more time copying logs, restarting services, or patching systems than building new features—when the same steps are performed every week, every release, every incident, yet remain error-prone and forgotten. This is the work nobody should be doing: the glue between systems, the routine that is never documented, the task that is never “done.”

Identifying Toil

Toil is work that is manual, repetitive, and not easily scalable. It is work that someone must do regularly but which does not advance the system’s core functionality. To identify toil, teams use a combination of direct observation, process mapping, and signal-based detection.

Start by logging all tasks performed over a two-week period. Use journalctl to capture systemd unit activity, dmesg for kernel events, and pg_log for PostgreSQL transaction logs. Then, query the system for recurring patterns:

journalctl --since "2 weeks ago" \
           --unit="nginx.service" \
           --unit="app.service" \
           --unit="postgres.service" \
           --grep="Started" \
           --output=short-precise

Look for entries like:

Apr 05 08:12:44 prod-01 systemd[1]: Started nginx.service.
Apr 05 08:13:11 prod-01 systemd[1]: Started app.service.
Apr 06 09:05:22 prod-01 systemd[1]: Started nginx.service.
Apr 06 09:10:45 prod-01 systemd[1]: Started app.service.

A task is toil if it occurs more than once per week and requires more than 15 minutes of effort, with no clear automation. Common signals include:

• Manual deployment steps (e.g., scp, rsync, systemctl restart, nginx -s reload)
• Incident triage tasks (e.g., checking top, htop, lsof, netstat, curl, tail -f)
• Configuration drift (e.g., manually editing /etc/nginx/sites-enabled/ on each server)
• Backup verification (e.g., ls /backups, md5sum, tar -tvf, find -mtime -1)
• Certificate renewal (e.g., openssl x509 -in cert.pem -text -noout, certbot renew)

Toil is revealed when a team discovers that 40% of their time is spent on tasks that could be fully automated with a few hours of scripting.

Automating Manual Tasks

Once toil is identified, the next step is to automate the task so it runs unattended, with consistent results. The automation must be:

• Idempotent: running multiple times produces the same result.
• Declarative: the desired state is defined, not the sequence of actions.
• Observable: output is logged, errors are tracked, and results are visible.

For a typical manual task like “deploy a new version of the application,” the automation starts with a script that packages, transfers, and activates the release.

#!/bin/bash
# deploy.sh

set -euo pipefail

VERSION="$1"
DEPLOY_DIR="/opt/apps/releases/$VERSION"

# Build the application
make build

# Create release directory
mkdir -p "$DEPLOY_DIR"

# Copy assets
cp -r dist/ "$DEPLOY_DIR/"

# Create symlink to current
ln -nsf "$DEPLOY_DIR" /opt/apps/current

# Update systemd unit
cat > /etc/systemd/system/app.service << EOF
[Unit]
Description=My App
After=network.target

[Service]
Type=simple
User=app
WorkingDirectory=$DEPLOY_DIR
ExecStart=/usr/bin/node index.js
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd, start service
systemctl daemon-reload
systemctl restart app.service

# Log success
echo "Deployed $VERSION" >> /var/log/deployments.log

The script is then triggered via cron:

# Run deployment at 2:15 AM every day
15 2 * * * /opt/scripts/deploy.sh 1.2.3

This simple automation eliminates toil, but only if it runs reliably. The real challenge is making it self-healing.

Making Automation Self-Healing

Automation fails silently. A deployment script may run but not update the current symlink. A log file may be written, but the system does not notice when it stops. Self-healing automation includes:

• Pre-execution checks: verify prerequisites before running.
• Post-execution verification: confirm the result.
• Error handling and retry: detect and recover from failure.
• Notification: alert when automation runs or fails.

A self-healing deployment script includes checks for service health and configuration:

#!/bin/bash
# deploy.sh with self-healing

set -euo pipefail

VERSION="$1"

# Check preconditions
if ! command -v make &>/dev/null; then
    echo "Error: make is not installed" >&2
    exit 1
fi

if ! systemctl is-active --quiet app.service; then
    echo "Error: app.service is not running. Starting..." >&2
    systemctl start app.service
fi

# Deploy as before...
make build
mkdir -p "/opt/apps/releases/$VERSION"
cp -r dist/ "/opt/apps/releases/$VERSION"
ln -nsf "/opt/apps/releases/$VERSION" /opt/apps/current

# Update systemd unit
cat > /etc/systemd/system/app.service << EOF
[Unit]
Description=My App
After=network.target

[Service]
Type=simple
User=app
WorkingDirectory=/opt/apps/releases/$VERSION
ExecStart=/usr/bin/node index.js
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl restart app.service

# Verify deployment
if ! curl -f http://localhost:3000/health; then
    echo "Error: app is not responsive after deployment" >&2
    exit 1
fi

# Check logs for errors
if grep -q "ERROR" /opt/apps/current/logs/app.log; then
    echo "Warning: errors found in app.log" >&2
fi

# Log success
echo "Deployed $VERSION" >> /var/log/deployments.log

This script now self-heals by:

• Failing fast if make is missing.
• Starting the app service if it’s down.
• Verifying the service is reachable via HTTP.
• Checking logs for known error patterns.

Handling Common Failure Modes

The most common failure mode in toil automation is partial success—a task completes but not all parts do. For example, a deployment script updates the code but forgets to reload nginx.

This happens when:

• Dependencies are not in sync: the app service runs, but nginx is still serving old code.
• Configuration drift: a new config is applied, but systemctl does not reload.
• Timing issues: the service starts before the database is ready.

To detect partial success, use health checks that validate the entire system after automation runs.

# health-check.sh
set -euo pipefail

# Check app service
if ! systemctl is-active --quiet app.service; then
    echo "App service is down" >&2
    exit 1
fi

# Check HTTP endpoint
if ! curl -f http://localhost:3000/health; then
    echo "App health check failed" >&2
    exit 1
fi

# Check database
if ! psql -c "SELECT 1" -d app_production; then
    echo "Database is unreachable" >&2
    exit 1
fi

# Check file existence
if ! [ -f /opt/apps/current/index.html ]; then
    echo "Index file missing in current release" >&2
    exit 1
fi

echo "All health checks passed"
exit 0

This script is run as a post-deployment task:

# Run health check after every deployment
0 3 * * * /opt/scripts/health-check.sh

If any check fails, the system sends an alert via mail or curl to a webhook:

# Send alert on health check failure
if ! /opt/scripts/health-check.sh; then
    curl -X POST \
         -H "Content-Type: application/json" \
         -d '{"message": "Deployment health check failed", "timestamp": "'"$(date)"'"}' \
         https://hooks.example.com/alerts
fi

The alert includes the output of journalctl and systemctl status:

journalctl --since "1 hour ago" --unit="app.service" --output=short-precise
systemctl status app.service --no-pager

Automating the Automation

Toil elimination is not a one-time task. It must be maintained. The automation itself must be automated.

This includes:

• Automating the automation script’s execution
• Automating versioning and release of the automation itself
• Automating updates to automation configuration

Use git to version the automation scripts:

# Track deployment scripts in git
git init
git add deploy.sh health-check.sh
git commit -m "Add deployment automation"
git push origin main

Then use git hooks to run the health check on every commit:

# .git/hooks/post-commit
#!/bin/bash
./health-check.sh

Automate the release of automation via a makefile:

.PHONY: deploy release

deploy:
	./deploy.sh $(VERSION)

release: deploy
	git tag -a v$(VERSION) -m "Release $(VERSION)"
	git push origin main --tags

Now, releasing a new version of the automation is a single command:

make VERSION=1.2.4 release

This command:

• Runs the deployment script
• Tags the repository
• Pushes the tag to the remote

The automation is now self-documenting, self-updating, and self-verifying.

The Unseen Traps of Toil Automation

Even with good automation, several traps emerge in production:

• The forgotten set -euo pipefail: without it, a script fails silently on a missing variable or command.
• The untested systemctl start app.service: the service may fail to start due to a missing environment variable or missing directory, but the script does not log the error.
• The misconfigured cron job: a crontab entry is added but not enabled; the systemd timer does not run.
• The unmonitored log file: logs are written, but the system does not check for new entries or parse errors.
• The stale nginx configuration: a new app.service is created, but nginx is not reloaded, so old code is served.

These traps are detected by:

• Adding set -x to scripts for detailed tracing.
• Using journalctl --unit=automation.timer to monitor when automation runs.
• Setting up logrotate for automation logs.
• Adding systemd timers to run scripts on a schedule instead of cron.

# /etc/systemd/system/automation.timer
[Unit]
Description=Run automation tasks daily
Requires=automation.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/automation.service
[Unit]
Description=Run daily automation tasks
After=network.target

[Service]
Type=oneshot
ExecStart=/opt/scripts/daily-automation.sh
User=automation
Group=automation
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

The automation now runs daily, logs to journalctl, and restarts if any task fails.

Conclusion

Toil elimination is not about replacing manual work with scripts. It is about making work invisible—so that engineers no longer need to remember to do it. The automation runs, the system verifies itself, and failures are detected before they cause incidents. The result is a system that runs with minimal attention: a system that does not need to be managed, only observed.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.