Build ChatOps workflows that let teams deploy, monitor, and respond to incidents directly from Slack or Discord. Covers bot architecture, command patterns, approval workflows, audit logging, and the cultural shift from runbooks to chat-driven operations.
ChatOps: Automating Operations Through Chat
ChatOps transforms chat platforms into the operational command center of engineering teams. When a system alerts, a service fails, or a deployment is due, teams respond not just with messages but with actions—automated, tracked, and repeatable—directly in the chat stream. It matters when operations must be collaborative, fast, and visible: when incidents require cross-team coordination, when on-call engineers need to act on alerts with minimal context, or when release pipelines must be reviewed and approved through chat.
Every chat message is a command. A team types /deploy production, and the system responds with status, logs, and a progress bar. This is not a feature—it is the core workflow.
The command format is simple: a leading slash (/) followed by a verb and optional arguments. The system parses the message using a standard pattern: /<command> [arg1=value] [arg2=value] ....
For example, to restart a service named api-gateway on the staging environment:
/deploy staging service=api-gateway revision=abc123The system parses the command and dispatches it to a queue or directly to a runner. The command is processed asynchronously, and the result is posted back to the chat thread.
Commands may accept flags with multiple values or subcommands:
/monitor app=web-server duration=30s threshold=95%The system can validate arguments at the time of parsing. Missing required arguments result in an immediate error:
Error: Missing required argument 'app'
Usage: /monitor app=<name> [duration=<time>] [threshold=<percent>]Failure to parse a command is logged, and the original message is updated to include the error.
ChatOps systems respond to events with structured feedback. When a service fails, the system posts a message with a rich payload: a summary, a link to logs, and actionable buttons.
The response is sent as a JSON object over HTTP to the chat platform’s webhook endpoint. The following message is posted to a Slack channel:
{
"text": "🔥 Service `auth-service` is down",
"attachments": [
{
"title": "Health Check",
"text": "Last 5 minutes: 98% uptime, 47ms avg latency",
"fields": [
{ "title": "Status", "value": "Critical", "short": true },
{ "title": "Last Check", "value": "2024-04-12T14:32:11Z", "short": true },
{ "title": "Error", "value": "Connection timeout to database", "short": false }
],
"color": "#ff0000",
"actions": [
{
"name": "retry",
"text": "Retry Now",
"type": "button",
"url": "/api/commands/retry?service=auth-service"
},
{
"name": "investigate",
"text": "Open Logs",
"type": "button",
"url": "/logs?service=auth-service&since=2024-04-12T14:00Z"
}
]
}
],
"channel": "incident-ops"
}The actions array defines interactive buttons. Clicking a button triggers a new HTTP POST to the specified URL, often with a payload field containing the action context.
The system must handle the case where a button click fails to deliver. It logs the failure and retries the request using exponential backoff: first attempt after 1 second, then 2, 4, 8, and so on. If the final attempt fails, the system marks the action as “failed” and posts a follow-up message:
⚠️ Action 'retry' failed: HTTP 503, service unavailableChatOps enables complex workflows through command chains. A single command can trigger a sequence of steps, each executed in order, with shared context.
To deploy a new version of a service, a team runs:
/deploy productionInternally, the system runs a workflow defined as a sequence of steps:
steps:
- name: "validate"
command: "validate-deployment"
args:
environment: "production"
service: "api-gateway"
revision: "abc123"
timeout: "60s"
retry: 3
- name: "migrate-db"
command: "migrate-database"
args:
schema: "api-gateway"
version: "v2.1"
depends_on: ["validate"]
- name: "rollout"
command: "rollout-service"
args:
service: "api-gateway"
strategy: "blue-green"
weight: 0.5
depends_on: ["migrate-db"]
- name: "verify"
command: "verify-service"
args:
service: "api-gateway"
endpoints: ["GET /health", "GET /metrics"]
depends_on: ["rollout"]
timeout: "120s"Each step is executed in a runner process. The runner starts with a shared environment: the current user, the chat thread ID, and a context object passed to each step.
The runner communicates with the chat platform using a standard API: it can post messages, update existing messages, and add reactions.
When a step fails, the runner posts an error message and updates the parent message with a status update:
✅ validate: Passed (12.3s)
🟡 migrate-db: Running...If a step times out, the system marks it as timeout, sends a follow-up message, and optionally triggers a new command:
⏰ Step 'migrate-db' timed out after 90s
➡️ /notify ops "Database migration stalled; manually investigate"The workflow is idempotent: the same command can be re-run, and the system checks if a step is already complete. If so, it skips the step and continues.
ChatOps maintains state across messages. When a user starts a workflow, the system tracks it in a database with a unique identifier.
A message like:
/start-cleanupcreates a new workflow instance with a unique workflow_id, stored in a table:
CREATE TABLE chat_workflow (
id UUID PRIMARY KEY,
command TEXT NOT NULL,
status TEXT CHECK (status IN ('pending', 'running', 'completed', 'failed')),
context JSONB,
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
chat_thread_id TEXT,
created_by TEXT
);The system returns a confirmation:
✅ Started cleanup workflow #7a3b2c8e
Use /status 7a3b2c8e to check progress.The context column stores the state of the workflow. After each step, the runner updates the context with results, variables, and intermediate data.
When a user runs /status 7a3b2c8e, the system queries the database and returns a summary:
{
"workflow_id": "7a3b2c8e",
"status": "running",
"steps": [
{
"name": "cleanup-old-logs",
"status": "completed",
"duration": "24.1s",
"output": {
"files_deleted": 142,
"total_size": "3.2 GB"
}
},
{
"name": "prune-docker-images",
"status": "running",
"progress": "67%",
"last_updated": "2024-04-12T15:22:03Z"
}
],
"actions": [
{
"name": "retry",
"text": "Retry prune-docker-images",
"url": "/api/workflows/7a3b2c8e/retry?step=prune-docker-images"
},
{
"name": "cancel",
"text": "Cancel workflow",
"url": "/api/workflows/7a3b2c8e/cancel"
}
]
}The state is persisted and available across multiple chat sessions and users. A user can return to a workflow days later and see its full history.
ChatOps systems must handle silent failures. A command may succeed at the runner level but fail to update the chat thread.
Common failure modes include:
/deploy prod env=staging, but the system expects environment. The command is processed, but the environment field is null. The system logs a warning: 📌 Warning: Unknown argument 'env' in /deploy; using 'environment' instead.status field and skips the step if the dependency is completed, but fails if it is failed.revision field, but the context has rev instead. The system logs a mismatch and attempts to resolve it by mapping keys: {
"error": "Missing context field 'revision'",
"suggested": [
{ "field": "rev", "value": "abc123" },
{ "field": "revision", "value": "abc123" }
],
"action": "use rev as revision"
}When a step fails, the system updates the chat message with a detailed error log. The log is sent as a separate message or embedded in the attachment:
{
"text": "❌ Step 'migrate-db' failed",
"attachments": [
{
"title": "Error Log",
"text": "SQLSTATE[HY000]: General error: 1213 Deadlock found when trying to get lock",
"fields": [
{ "title": "Step", "value": "migrate-db", "short": true },
{ "title": "Duration", "value": "94.2s", "short": true },
{ "title": "Exit Code", "value": "1", "short": true }
],
"color": "#b30000",
"footer": "Run at 2024-04-12T14:40:21Z",
"actions": [
{
"name": "retry",
"text": "Retry Step",
"type": "button",
"url": "/api/steps/migrate-db/retry?run=1234"
}
]
}
]
}The system also supports error escalation: if a step fails, and the last two attempts were also failures, the system posts a message to a dedicated alerts channel and notifies on-call engineers via email or SMS.
ChatOps is not limited to chat. It integrates with external tools using standard protocols.
To fetch logs from a service, the system calls curl with a standard API:
curl -X GET \
-H "Authorization: Bearer $API_TOKEN" \
-H "Accept: application/json" \
"https://logs.example.com/v1/services/api-gateway?since=2024-04-12T14:00:00Z&limit=100"The response is parsed and formatted into a chat message:
📋 Recent logs from api-gateway (last 60 minutes):
2024-04-12T14:58:12Z | ERROR | Database connection failed
2024-04-12T14:57:45Z | WARNING | High latency: 380ms
2024-04-12T14:56:33Z | INFO | User login successfulThe system can also execute commands via kubectl:
kubectl get pods -n production -l app=api-gateway -o jsonThe output is parsed and formatted as a table in the chat message.
For database interactions, the system uses psql directly:
SELECT
id, status, error, started_at
FROM deployments
WHERE environment = 'staging'
ORDER BY started_at DESC
LIMIT 5;The result is rendered as a markdown table:
| id | status | error | started_at |
|---|---|---|---|
| d123 | completed | null | 2024-04-12 14:30:00 |
| d122 | failed | Timeout after 60s | 2024-04-12 14:00:00 |
The system caches results and invalidates them when new data is available.
/scale app=web-server nodes=4 may be parsed correctly, but if the system expects node_count, the nodes field is silently ignored.service field, but the context is passed as target_service instead. The system should warn about missing or unexpected keys.status of error, but the message says failed. The system should standardize on a set of status codes: pending, running, success, failed, timeout, cancelled.message_id field in the payload./deploy production twice, the system may start two workflows. The system should deduplicate commands based on command + arguments + environment.ChatOps is not a tool—it is a pattern. It is the act of turning every message into a command, every response into a workflow, and every incident into a shared history.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy