Network Security Monitoring

Production engineering guide for network security monitoring covering patterns, implementation strategies, and operational best practices.

Network Security Monitoring (NSM) is the continuous process of collecting, analysing, and acting on network traffic and metadata to detect, investigate, and respond to security threats. It matters when network-based attacks—lateral movement, command-and-control, data exfiltration, or anomaly detection—are critical, particularly in environments with dynamic workloads, multiple cloud providers, or high-volume traffic. NSM is not a one-time audit; it is the backbone of real-time threat visibility.

Ingesting Network Data

The first step is to gather raw network data from sources across the infrastructure. This includes flow records, packet captures, and metadata from firewalls, load balancers, and network devices. The most common ingestion mechanism is NetFlow, sFlow, or IPFIX, but all are built on the same principles: packets are sampled and exported with metadata like source/destination IPs, ports, protocol, and byte counts.

To collect NetFlow data, configure network devices to send records to a central collector. On a Cisco router, this requires:

ip flow-export destination 192.168.1.100 2055
ip flow-export version 9
ip flow-export template option-cache timeout 300
ip flow-export delay 30

The collector listens on UDP port 2055 and parses the flow records. A typical flow record contains:

A common failure mode is flow record loss due to UDP packet drops. Without explicit acknowledgments, missing records leave blind spots. To mitigate, configure flow exporters to use multiple UDP ports, or use TCP-based flow protocols like NetFlow over TCP or IPFIX with TCP transport.

For packet-level visibility, deploy sensors that capture full packets. Tools like Zeek (formerly Bro) or Suricata are standard. Zeek generates structured logs in a consistent format, with each event in a JSON object:

{
  "ts": 1692345600.123456,
  "uid": "C123456789",
  "id.orig_h": "10.0.1.45",
  "id.orig_p": 52012,
  "id.resp_h": "192.168.2.12",
  "id.resp_p": 443,
  "proto": "tcp",
  "duration": 30.000,
  "orig_bytes": 12456,
  "resp_bytes": 23450,
  "conn_state": "SF",
  "missed_bytes": 0,
  "history": "ShDd",
  "orig_pkts": 18,
  "resp_pkts": 25,
  "orig_ip_bytes": 14500,
  "resp_ip_bytes": 25000
}

A silent failure occurs when time synchronization drifts between sensors and the central analysis system. If the sensor clock is off by 5 seconds, event correlation across systems becomes unreliable. Ensure all sensors use NTP with a common time source and enable time_sync in Zeek’s configuration:

redef time_sync = T;

Correlating Events Across the Network

With data ingested, the next task is to correlate events across multiple sources. This involves merging flow records, packet logs, and metadata from firewalls, load balancers, and security appliances.

The most effective way to correlate is to use a common identifier—usually a unique transaction ID or a combination of source/destination IP and port. For example, when a client connects to a web server via HTTPS, a single TCP connection can be traced across multiple systems:

To merge these, extract the 5-tuple (src_ip, dst_ip, src_port, dst_port, protocol) and use it as a key. A common mistake is using only src_ip and dst_ip while ignoring port numbers. This leads to false positives when multiple services share the same IP.

A frequent failure is mismatched timestamps. If the firewall logs use local time and Zeek uses UTC, events appear out of order. To correct this, convert all timestamps to UTC and normalize time zones:


TZ=America/New_York date -d "2023-08-15 14:30:00" +"%Y-%m-%dT%H:%M:%SZ"
# Output: 2023-08-15T18:30:00Z

For large-scale correlation, use a message queue like Kafka or RabbitMQ. A typical pipeline:

  1. Sensors write flow and packet logs to Kafka topics.
  2. A consumer service reads from network.flow and network.packet topics.
  3. It joins records using the 5-tuple and emits a unified event stream.
  4. Events are stored in a time-series database such as InfluxDB or TimescaleDB.
-- In TimescaleDB, create a hypertable for network events
CREATE TABLE network_events (
  time TIMESTAMPTZ NOT NULL,
  src_ip INET,
  dst_ip INET,
  src_port INT,
  dst_port INT,
  protocol INT,
  flow_id TEXT,
  event_type TEXT,
  payload JSONB
);

SELECT create_hypertable('network_events', 'time');

A subtle failure occurs when the 5-tuple does not uniquely identify a connection. If two clients connect to the same server on the same port simultaneously, their flows overlap. To resolve this, include the uid field from Zeek or use a composite key: (src_ip, src_port, dst_ip, dst_port, protocol, flow_id).

Detecting Anomalies and Threats

Once events are correlated, the next step is to detect anomalies and known threats. This involves defining rules, thresholds, and patterns.

A basic anomaly detection rule is identifying hosts that generate unusually high outbound traffic. For example, a server in the database subnet sends 100 MB of data in 5 minutes—30 times the average.

To detect this, run a query over the past hour of flow data:

-- In SQL, compute outbound traffic per host
SELECT
  src_ip,
  SUM(bytes) AS total_outbound,
  AVG(bytes) AS avg_outbound,
  STDDEV(bytes) AS stddev_outbound,
  COUNT(*) AS flow_count
FROM network_flow
WHERE time >= NOW() - INTERVAL '1 hour'
  AND protocol = 6  -- TCP
  AND src_port >= 1024
GROUP BY src_ip
HAVING SUM(bytes) > 3 * AVG(bytes) + 2 * STDDEV(bytes);

The rule triggers when a host’s outbound traffic exceeds three standard deviations above its mean. A common error is using a fixed threshold (e.g., 50 MB) instead of a statistical one. This fails when traffic patterns change seasonally or due to new application deployments.

For known threats, use signature-based detection. Suricata, for example, supports YARA-like rules in a simple text format:

# signature.yaml
- rule:
    id: 10001
    name: "Suspicious Outbound DNS over UDP"
    priority: 1
    metadata:
      event_type: "network"
      category: "external"
    condition:
      - "flow.src_port == 53"
      - "flow.dst_port == 53"
      - "flow.protocol == 17"  # UDP
      - "flow.bytes > 1000"
      - "flow.packets > 1"
    alert:
      message: "High-volume DNS query over UDP"
      severity: "high"

Suricata applies these rules to incoming packets and emits alerts. A silent failure is misconfigured flow fields: if the flow.src_port is missing in the packet, the rule fails. Ensure that the flow engine is active and that the rule is loaded correctly.

Responding to Detected Events

After detection, the system must respond—either by alerting, automating actions, or triggering incident response workflows.

A common response is sending alerts to a centralized dashboard. Tools like Grafana or Kibana can ingest events and visualize them. Grafana queries the time-series database:

{
  "datasource": "influxdb",
  "query": {
    "measurement": "network_events",
    "tags": [
      { "key": "event_type", "value": "alert" }
    ],
    "groupBy": [
      { "type": "time", "unit": "minute" }
    ],
    "select": [
      [ { "type": "field", "value": "count" } ]
    ]
  }
}

When an alert is triggered, the system can automatically execute a response action. For example, when a host is detected as a command-and-control server, the system can:

  1. Query the host’s IP in a threat intelligence feed via HTTP.
  2. Enrich the event with additional data.
  3. Block the host’s IP at the firewall.

The response is typically implemented as a script or service. A minimal example using curl and jq:

#!/bin/bash
# response.sh
set -euo pipefail

IP="$1"
URL="https://api.threatintel.example.com/v1/ip/$IP"

# Fetch threat intelligence
THREAT_DATA=$(curl -s "$URL")

# Extract indicators
C2_DOMAIN=$(echo "$THREAT_DATA" | jq -r '.c2_domains[0]')
THREAT_SCORE=$(echo "$THREAT_DATA" | jq -r '.threat_score')

# Enrich event
echo "{
  \"event_type\": \"c2_detected\",
  \"source_ip\": \"$IP\",
  \"c2_domain\": \"$C2_DOMAIN\",
  \"threat_score\": $THREAT_SCORE,
  \"timestamp\": \"$(date -Iseconds)\"
}" > /var/log/network/alerts/c2_$IP.json

# Apply firewall rule
sudo iptables -A OUTPUT -d "$IP" -j ACCEPT
sudo iptables -A INPUT -s "$IP" -j ACCEPT
sudo iptables -A OUTPUT -d "$IP" -j DROP
sudo iptables -A INPUT -s "$IP" -j DROP

# Save rule set
sudo iptables-save > /etc/iptables/rules.v4

A sharp edge: the response script runs but does not handle network failures. If the threat intelligence API is unreachable, the script fails silently. To fix, add retry logic and logging:

# Add retries to curl
curl -s --retry 3 --retry-delay 2 "$URL"

Maintaining and Tuning the System

Finally, NSM systems require ongoing maintenance. The most common issues are data loss, performance degradation, and rule drift.

Data loss occurs when ingestion pipelines fail. A Kafka consumer that processes flow records may miss messages if it crashes during processing. To ensure reliability, use message acknowledgments and idempotent processing:

A subtle failure is unprocessed messages due to mismatched time zones. A consumer reads messages from Kafka with timestamps in UTC, but the processing script assumes local time. This leads to missed events in the evening hours.

Performance degradation happens when the volume of data exceeds processing capacity. A common sign is increasing lag in the Kafka consumer. To diagnose:

# Check Kafka consumer lag
kafka-consumer-groups.sh --bootstrap-server kafka.example.com:9092 \
  --describe --group network-monitoring

The output shows CURRENT-OFFSET, LOG-END-OFFSET, and LAG for each consumer. A lag of 10,000 messages indicates the system is behind.

To tune, adjust the fetch.min.bytes and fetch.max.wait.ms settings in Kafka. Larger fetches reduce network overhead, while longer waits reduce latency.

Another failure mode is rule drift: over time, detection rules become outdated. A rule that flags high outbound traffic now misses a new application that sends 500 MB per hour. To address this, schedule regular reviews and use a versioned rule repository:

# rules/v1.0.yaml
- rule:
    id: 10001
    name: "High outbound traffic"
    description: "Detect hosts sending more than 3σ above average"
    version: "1.0"
    updated: "2023-08-15"
    conditions:
      - "flow.bytes > 3 * avg_flow_bytes + 2 * stddev_flow_bytes"
      - "flow.duration > 30"

A script can compare current rules with a baseline and report changes:

# Compare rule versions
git diff --name-status HEAD~1 HEAD

When a new rule version is merged, the system applies it automatically, ensuring that detection logic evolves with the network.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.