Security Logging Siem

Production engineering guide for security logging siem covering patterns, implementation strategies, and operational best practices.

Security Logging and SIEM (Security Information and Event Management) is the practice of collecting, normalizing, enriching, and analyzing security-relevant events from systems across an environment to detect threats, troubleshoot incidents, and meet compliance requirements. It matters when a security team must answer: *Where did this attack originate? How long did it take to detect? What systems were compromised?* The foundation is a centralized, real-time view of logs, with the ability to correlate events across hosts, applications, network devices, and cloud services.

Log Ingestion and Normalization

Log ingestion is the process of receiving raw log data from sources and preparing it for analysis. The most common ingestion method uses syslog or HTTP, with structured formats like JSON or CEF (Common Event Format) as the standard.

The core pipeline begins with a log producer emitting events in a consistent schema. For example, a Linux system using systemd-journald exports logs via journalctl --output=json:

{
  "MESSAGE": "User bob logged in via SSH",
  "SYSLOG_IDENTIFIER": "sshd",
  "UID": 1001,
  "PID": 1234,
  "SYSLOG_FACILITY": 2,
  "PRIORITY": 6,
  "HOSTNAME": "web-01.example.com",
  "TIMESTAMP": "2024-05-14T12:03:45.678Z"
}

This JSON is shipped to a central log collector via rsyslog or fluentd. A fluentd configuration file (fluent.conf) defines an input and output:

<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<match **>
  @type copy
  <store>
    @type file
    path /var/log/siem/ingested
    format json
  </store>
  <store>
    @type elasticsearch
    host elasticsearch.example.com
    port 9200
    index_name siem-logs-${time_slice}
    time_key TIMESTAMP
    time_format %Y-%m-%dT%H:%M:%S.%LZ
    include_time_key true
    flush_interval 10s
  </store>
</match>

The time_key field is critical: it defines the timestamp used for indexing and time-based queries. If TIMESTAMP is not formatted as ISO 8601 with subseconds and timezone, logs arrive out of order or with incorrect time alignment.

The syslog protocol uses UDP or TCP to stream logs in a line-based format:

<14>1 2024-05-14T12:03:45.678Z web-01.example.com sshd 1234 - User bob logged in via SSH

The priority field <14> encodes facility (2) and severity (6). This format is widely supported but lacks schema. To normalize, a SIEM ingests raw logs and applies a parser rule:

Logs may be ingested from cloud services via APIs. AWS CloudTrail emits logs in JSON format, and aws logs put-log-events expects:

{
  "logGroupName": "/aws/ecs/service",
  "logStreamName": "web-service",
  "logEvents": [
    {
      "timestamp": 1715712225000,
      "message": "Container started: 9a2c3b1d"
    }
  ]
}

The timestamp is in milliseconds since Unix epoch. If the SIEM expects microseconds, events are off by a factor of 1000, causing time drift across systems.

Event Correlation and Enrichment

Once logs are ingested, the SIEM correlates events from different sources using common fields. The most critical fields are timestamp, source, event_type, host, user, and ip.

A common failure occurs when user is not consistently populated. An SSH login may record USER in the message but not in a user field. A SIEM rule must extract user from MESSAGE using a regex:

user: (?P<user>\w+)

Without this, a login from alice is not linked to her account in Active Directory.

Enrichment adds context to events. A SIEM enriches IP addresses by querying a threat intelligence feed:

{
  "ip": "192.168.1.10",
  "country": "US",
  "city": "San Francisco",
  "organization": "Cloudflare, Inc.",
  "is_anonymous": false,
  "is_proxy": true,
  "risk_score": 73,
  "malware": [
    "Cobalt Strike",
    "Mimikatz"
  ]
}

The SIEM merges this into the original event, creating a new field threat.intelligence for use in queries and dashboards.

Another enrichment pattern uses a local database. A PostgreSQL table user_attributes contains user, department, role, manager, onboarded_at.

A SIEM query enriches every event with user context:

SELECT e.*, ua.department, ua.role
FROM events e
JOIN user_attributes ua ON e.user = ua.user
WHERE e.timestamp >= now() - interval '1 hour'

If the join fails due to case mismatch (user vs USER), the enrichment is incomplete. The query must explicitly normalize case:

ON lower(e.user) = lower(ua.user)

Querying and Alerting

Querying is the primary way to explore logs and detect anomalies. The most common query language is a SQL-like syntax with built-in functions and time-based operators.

A typical query to find failed SSH logins in the last hour:

FROM events
WHERE
  source = 'sshd'
  AND event_type = 'auth'
  AND result = 'failed'
  AND timestamp >= now() - interval '1 hour'
ORDER BY timestamp DESC
LIMIT 100

The interval syntax uses standard SQL syntax with units: minute, hour, day, week, month.

An alert triggers when a condition is met. A high-fidelity alert uses a threshold:

Alert: Failed SSH Logins (5+ in 10 minutes)
Trigger condition:
  COUNT(events) WHERE
    source = 'sshd'
    AND result = 'failed'
    AND timestamp >= now() - interval '10 minutes'
  >= 5

The alert sends an email with the top 10 events and a summary chart.

A more complex alert uses temporal correlation: *a user logs in from a new IP, followed by a database login within 5 minutes.*

WITH logins AS (
  SELECT user, ip, timestamp
  FROM events
  WHERE source = 'sshd'
    AND result = 'success'
),
db_access AS (
  SELECT user, ip, timestamp
  FROM events
  WHERE source = 'db'
    AND event_type = 'login'
)
SELECT l1.user, l1.ip AS login_ip, d1.ip AS db_ip, l1.timestamp AS login_time, d1.timestamp AS db_time
FROM logins l1
JOIN db_access d1 ON l1.user = d1.user
WHERE d1.timestamp BETWEEN l1.timestamp AND l1.timestamp + interval '5 minutes'
  AND l1.ip <> d1.ip
ORDER BY l1.timestamp DESC

This query detects lateral movement: users accessing services from unfamiliar locations.

Alerts may fail silently if thresholds are misconfigured. For example, a daily alert for failed logins expects 100 events per day, but the data source logs only 30. The alert fires every day but with only 10% of expected events. The alert condition should include AND COUNT(events) > 10 to catch under-populated days.

Dashboards and Incident Response

Dashboards visualize event flows and provide operational insight. A common dashboard includes:

A dashboard query for the top users:

SELECT
  user,
  COUNT(*) AS failed_logins,
  AVG(timestamp - first_login) AS avg_time_to_first_login,
  ARRAY_AGG(DISTINCT ip) AS login_ips
FROM events
WHERE
  source = 'sshd'
  AND result = 'failed'
  AND timestamp >= now() - interval '7 days'
GROUP BY user
ORDER BY failed_logins DESC
LIMIT 10

The ARRAY_AGG function collects all IPs used by a user. The dashboard renders this as a list with a bar chart.

Incident response begins with a new alert. The responder opens the event details and sees:

They expand the timeline and notice a pattern: multiple failed login events from 192.168.1.10, followed by successful login from 192.168.1.11, and then failed from 192.168.1.12. This suggests a credential stuffing attack.

They create a new investigation case and run a query to extract all events from 192.168.1.10 over the past 24 hours:

FROM events
WHERE ip = '192.168.1.10'
  AND timestamp >= now() - interval '24 hours'
ORDER BY timestamp

They save this query as a reusable template for future investigations.

Common Failure Modes

A SIEM fails silently when log ingestion is misconfigured. The most common failure: logs arrive but are not indexed or searchable.

Real-World Example

A security engineer investigates a phishing attack. The alert triggers: *User [email protected] logged in from IP 192.168.1.10 (new IP) within 10 minutes of a failed login.*

They open the event and find:

The engineer runs a query:

FROM events
WHERE
  user = '[email protected]'
  AND ip = '192.168.1.10'
  AND timestamp >= '2024-05-14T12:00:00Z'
  AND timestamp < '2024-05-14T12:15:00Z'
ORDER BY timestamp

They discover a sequence of events: login from a new IP, followed by API calls to a web service, and then a firewall block. They create a case and mark the event as phishing_attack with source=alert, priority=high, and assigned_to=security_team.

The SIEM now has a documented incident, with a timeline, associated events, and a resolution path.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.