Security Monitoring Siem

A Security Information and Event Management (SIEM) system collects log and event data from across an infrastructure, applies correlation logic, and surfaces alerts for investigation. It matters when a breach or misconfiguration spans multiple systems and cannot be detected from any single log source alone. This page is for engineers who need to get data in, tune detection logic, investigate alerts, and forward results—without guessing at vendor‑specific internals.

Getting data into the SIEM

The first step is ensuring event sources ship data reliably. Most SIEMs accept inputs via filebeat/otel collectors, syslog, or direct API ingestion. What follows are patterns that apply across platforms.

Shipping logs via collector

A typical filebeat input configuration looks like this:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/auth.log
    - /var/log/secure
  fields:
    source: sshd
  toward:
    - host: siem.example.com
      port: 5044
      ssl: true

If events fail to appear, first verify the collector can reach the intake endpoint:

curl -s https://siem.example.com:5044/health -o /dev/null -w "%{http_code}"

A 200 response means the intake endpoint is reachable; anything else indicates network or TLS configuration issues. A frequent silent failure is mismatched JSON field names—filebeat ships the raw line, and the SIEM’s parser expects specific keys (@timestamp, message, host). If those keys are absent, the event is dropped without an error entry in the collector log.

Direct API ingestion

Some environments bypass collectors and post events directly via HTTP. A minimal curl example:

curl -s -X POST "https://siem.example.com/api/v1/events" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"timestamp":"2024-05-15T10:23:45Z","source":"webserver","event_id":"evt-9912","severity":"high","description":"failed login"}'

The server responds 202 Accepted on success. If the token is expired or the JSON schema mismatches, the response is 401 Unauthorized or 400 Bad Request with a body like:

{"error":"invalid_event_schema","missing_fields":["@timestamp","severity"]}

A common gotcha: the Content-Type header must match what the SIEM expects. Omitting it often defaults to application/x-www-form-urlencoded, which causes the payload to be parsed as form fields rather than JSON, resulting in silently missing data.

Verifying ingest with psql

Many SIEM backends store raw events in a relational store for debugging. Query the most recent inbound events:

psql -h siem-db -U sdb_user -d sdb -c \
  "SELECT event_id, received_at, source, severity, raw_json FROM events \
   WHERE received_at >= NOW() - INTERVAL '5 minutes' ORDER BY received_at DESC LIMIT 10;"

If the query returns zero rows despite active shipping, check the SIEM’s index lifecycle management (ILM) policy. Some configurations delete events after a short retention window (e.g., 30 days), and if the SIEM was restarted or re‑indexed, recent events may have been purged before they could be queried.

Tuning correlation rules

Once data flows in, detection rules decide what becomes an alert. Rules vary in syntax but share common parameters that are frequently confused.

Rule anatomy (generic)

A typical rule expresses a condition over a time window:

 WHEN count of events where src_ip = $IP AND event_type = "auth_fail" > 5
       WITHIN 2 minutes
 THEN generate alert "brute_force_ssh"

The critical parameters are count, WITHIN, and the implicit time_window that the SIEM uses to bucket events. A frequent mistake is setting count: 5 with a WITHIN 1 minute window when the expected pattern spans 10 minutes. The rule fires after the first 5 events in any 1‑minute slice, producing alerts far earlier than intended.

Testing rules without alerting

Most SIEMs provide a “preview” or “dry‑run” mode. For a generic SIEM API:

curl -s -X POST "https://siem.example.com/api/v1/rules/test" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "rule_id": "brute_force_ssh",
        "condition": {
          "count": 5,
          "time_window": "2m"
        },
        "test_events": [
          {"timestamp":"2024-05-15T10:00:00Z","src_ip":"10.0.0.1","event_type":"auth_fail"},
          {"timestamp":"2024-05-15T10:00:30Z","src_ip":"10.0.0.1","event_type":"auth_fail"},
          {"timestamp":"2024-05-15T10:01:00Z","src_ip":"10.0.0.1","event_type":"auth_fail"},
          {"timestamp":"2024-05-15T10:01:30Z","src_ip":"10.0.0.1","event_type":"auth_fail"},
          {"timestamp":"2024-05-15T10:02:00Z","src_ip":"10.0.0.1","event_type":"auth_fail"}
        ]
      }'

The response includes a verdict field: alert if the condition fires, pass otherwise. This is the safest way to validate a rule before committing it to the active rule set.

Confused options

Two options that administrators routinely mix up are threshold and rate:

• threshold N fires once the total count across the window exceeds N.
• rate N per M fires when the event rate exceeds N occurrences per M time units.

If a rule intends to alert on more than 10 auth failures per hour, threshold 10 with a 1‑hour window works. But rate 10 per 1 minute would trigger after just 10 failures in any single minute, which is usually not the intent. The confusion compounds when both options appear in the same rule UI, and the documentation does not clearly label which time unit each uses.

Investigating alerts

When an alert fires, the engineer needs to reconstruct what happened around that moment. Concrete queries using curl and psql are the primary tools.

Fetching raw events around alert time

Assume alert ABC-7742 fired at 2024-05-15T14:32:11Z. Pull the raw events the SIEM correlated:

curl -s "https://siem.example.com/api/v1/events?alert_id=ABC-7742&start=2024-05-15T14:30:00Z&end=2024-05-15T14:35:00Z" \
  -H "Authorization: Bearer $TOKEN" | jq .

The JSON array contains every event the rule examined, with fields timestamp, src_ip, dest_ip, and the raw message string. If the array is empty, the alert may have been generated from cached state or from a different data source not retained in the events index.

Drilling into alert context via SQL

The alerts table often stores a JSON payload with metadata. Query it to see which rule fired, what correlated fields were present, and the severity escalation path:

psql -h siem-db -U sdb_user -d sdb -c \
  "SELECT alert_id, rule_name, severity, correlated_fields, first_seen, last_seen \
   FROM alerts WHERE alert_id = 'ABC-7742';"

Typical output columns:

alert_id	rule_name	severity	correlated_fields	first_seen	last_seen
ABC-7742	brute_force_ssh	high	{"src_ip":"10.0.0.1","failed_attempts":7}	2024-05-15T14:29:58Z	2024-05-15T14:32:11Z

If correlated_fields is null, the rule may have fired on a single event rather than a correlation, which is a different failure mode to investigate.

Common investigation dead ends

• Timestamp drift: Events from network devices may have timestamps offset by minutes or hours. Always verify received_at vs event.timestamp in the raw JSON.
• Field aliasing: Some collectors rename fields (e.g., source_ip → src). If the rule references source_ip but the event has src, the correlation silently fails.
• Data gaps: If the SIEM’s internal clock skews, the WITHIN window in a rule may never align with incoming events, causing the rule to never fire despite matching patterns.

Forwarding and exporting alerts

Alerts often need to reach ticketing systems, SOAR platforms, or dashboards. The mechanisms are generic across SIEMs.

Webhook forwarding

A typical webhook configuration key (generic pattern) looks like:

output.webhook.url: https://hooks.example.com/incident
output.webhook.method: POST
output.webhook.headers.Authorization: Bearer $TOKEN
output.webhook.payload_template: |
  {
    "alert_id": "{{alert_id}}",
    "rule": "{{rule_name}}",
    "severity": "{{severity}}",
    "src_ip": "{{src_ip}}"
  }

When an alert fires, the SIEM POSTs the rendered template to the URL. A frequent silent failure is TLS certificate mismatch—if the webhook endpoint uses a self‑signed cert and the SIEM is configured without insecure: true, the POST fails with x509: certificate signed by unknown authority. Adding insecure: true resolves the transport error but exposes the payload to interception; the proper fix is to deploy a valid certificate from a trusted CA.

API rate limits

Forwarding destinations often enforce rate limits. If the SIEM sends more than the allowed requests per second, subsequent webhooks receive 429 Too Many Requests. The SIEM’s back‑off logic (if configured) retries after a delay, but many default installations have no retry policy, resulting in dropped alerts. Monitor the forwarder’s error counter:

curl -s https://siem.example.com/api/v1/forwarder/stats \
  -H "Authorization: Bearer $TOKEN" | jq '.dropped_webhooks'

A rising count indicates the destination cannot keep up, and throttling the SIEM’s alert rate or increasing the destination’s concurrency is required.

Exporting to CSV for analysis

Occasionally an engineer needs a static snapshot for offline correlation. Most SIEMs provide an export endpoint:

curl -s -X POST "https://siem.example.com/api/v1/exports/csv" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"alert_ids":["ABC-7742","ABC-7743"],"fields":["timestamp","src_ip","dest_ip","rule_name"]}' \
  -o alerts.csv

The response is a CSV file with a header row and one line per alert‑event pair. If the alert_ids list contains an ID that doesn’t exist, the API returns 404 Not Found for that specific ID while processing the rest. Always validate IDs before bulk export.

Sharp edges you only learn after breaking production

• Index lifecycle purge: An ILM policy that deletes indices older than 90 days can erase events needed for post‑mortem analysis of a breach that occurred 85 days prior. Always verify retention covers the full investigation window.
• Correlation window edge cases: A rule with WITHIN 5 minutes may miss a pattern where the first and last events are exactly 5 minutes 1 second apart, depending on whether the SIEM uses inclusive or exclusive window bounds. Test with events at the boundary.
• Auth token rotation: SIEM webhook tokens often have a 24‑hour or 7‑day lifespan. If the token expires and no rotation script runs, all forwardings silently stop. The only symptom is a sudden drop in ticket creation from the SIEM. Automate token renewal via a cron job that calls the SIEM’s token refresh endpoint.
• Duplicate alert suppression: Some SIEMs deduplicate alerts based on src_ip + rule_name within a 1‑hour window. If a brute‑force attack spans multiple source IPs, each IP generates a separate alert, but within a single IP the second event is suppressed. This can give the illusion that the attack stopped when it actually continued from a different IP.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.