Production engineering guide for sre toil measurement covering patterns, implementation strategies, and operational best practices.
This page describes how to measure and track toil in software engineering operations, specifically focusing on quantifying the effort spent on manual, repetitive, or non-value-adding work. Toil measurement matters when establishing operational maturity, identifying automation opportunities, or benchmarking engineering team performance.
Toil is work that directly supports system reliability but does not add customer-facing value. Examples include responding to alerts, debugging production issues, performing routine maintenance, and implementing workarounds. The goal is to quantify how much time engineers spend on these activities relative to value-generating work.
Measurement requires tracking both the volume of toil tasks and their duration. Use a structured approach where each toil activity is logged with start/end timestamps, task description, and associated system components. This data enables calculating toil percentage as:
Toil Percentage = (Total Toil Time / Total Work Time) × 100Use a simple time-tracking mechanism for recording toil activities. Create a structured log format with these fields: timestamp, activity description, duration, affected system, and team member.
Example log entry:
2023-04-15T09:30:00Z - Investigating high CPU on API server
2023-04-15T10:15:00Z - Duration: 45 minutes
System: user-service
Engineer: jsmithThis can be stored in a flat-file database, CSV file, or simple JSON log. The key is consistency in format and regular updates.
Link toil activities directly to incident reports. Each incident should include a toil classification field that indicates whether the work was primarily operational (toil) or value-added (engineering).
For example, an incident report might show:
This classification helps distinguish between planned maintenance and unplanned reactive work.
Track toil by allocating time slots to different types of activities. Use a time-tracking tool that allows categorizing entries by type. Examples include:
Regularly audit these allocations to ensure they reflect actual work patterns.
Toil measurement helps identify automation candidates. Look for recurring activities that consume significant time. For example, if backup scripts require manual intervention 10 times per month, this represents a potential automation opportunity.
Use historical toil data to prioritize automation efforts. The most frequent or time-consuming tasks should be automated first.
To calculate toil metrics, collect data over at least one full work cycle (typically one week). Each toil entry should include:
Aggregate this data to produce:
Track changes to system configurations as part of toil measurement. Each configuration change that requires manual intervention counts as toil. Log these with the same structure used for other toil activities.
For example, if a database parameter change requires SSH access and restart:
Toil can be measured through service level indicators (SLIs). A common SLI is the frequency of manual interventions required to maintain system health.
Example SLI:
Manual Interventions Per Week = Total Manual Actions / Number of EngineersThis metric can be tracked in monitoring systems that log operational events.
Each change that requires manual approval or execution counts as toil. Track these changes separately from automated deployments.
Example tracking fields:
Present toil data in a standardized format. Create regular reports that show:
Reports should be shared with engineering managers and technical leads.
Look for patterns in toil activities to identify root causes. For example, if most toil occurs during specific times of day or involves certain systems, this suggests underlying problems.
Example pattern:
Peak Toil: 10am-2pm daily
Primary Systems: user-service, payment-apiTrack how often engineers must manually intervene in deployments. High deployment frequency with low automation often correlates with high toil.
Use deployment logs to identify manual steps that could be automated:
Measure alert-related toil by logging the number and duration of alert responses. This includes:
Set up baseline measurements before implementing automation efforts. Track toil for at least one month to establish a baseline.
Example baseline report:
Baseline Period: March 1-31, 2023
Average Toil Hours/Engineer: 24.5
Total Toil Hours: 147
Toil Percentage: 32%After implementing toil reduction measures, calculate improvements using the same metrics:
Use system observability tools to identify toil sources. Monitor logs, metrics, and traces to find recurring issues that cause manual intervention.
Example observability data points:
Include toil measurement in post-mortem processes. Document how much time was spent on reactive work versus proactive improvements.
Each retrospective should include:
Track how toil affects team velocity. High toil often correlates with reduced capacity for new feature development.
Example measurement:
Velocity = (Total Story Points Completed) / (Total Engineer Hours)
Toil Adjustment: Subtract toil hours from total available timeMeasure the time between incident detection and resolution. Longer response times often indicate more toil in the process.
Track these metrics:
Complex systems typically generate more toil. Track system complexity indicators:
Operational burden represents accumulated toil across all systems. Monitor this through:
Establish feedback loops where team members report toil experiences. This can be done through surveys, regular check-ins, or structured reporting.
Example feedback form:
1. What manual tasks take the most time?
2. Which systems require the most intervention?
3. What automation would reduce toil?
4. How often do you spend time on reactive work?Monitor how toil affects resource allocation decisions. High toil may indicate need for additional engineers or automation tools.
Example tracking:
Create dashboards that visualize toil metrics. These should include:
Example dashboard components:
- Weekly toil hours per engineer
- Toil distribution by system
- Toil trend line (last 12 weeks)
- Target vs. actual toil percentageLink toil measurement to overall engineering productivity. Track how toil reduction impacts:
Measure the value of toil reduction efforts. Calculate:
Example calculation:
Time Saved = 2 hours × 10 incidents = 20 hours
Value = 20 hours × $100/hour = $2,000Integrate toil measurement into continuous improvement processes. Regularly review and update toil definitions and measurement methods.
Example improvement cycle:
Recognize that toil measurement has limitations. Some manual work cannot be easily quantified, and human judgment affects classification accuracy.
Example limitation:
Ensure data quality through regular audits and validation processes. Check for:
Example audit process:
1. Review 10% of logged entries
2. Verify accuracy with team members
3. Correct errors
4. Update measurement procedures if neededThis page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy