Threat Intelligence Feed Integration Architecture

Threat intelligence feeds provide curated data about observed malicious activity, infrastructure, and threat actor tactics. Integrating them into your own security operations platform allows detection, response, and risk scoring to be informed by external context. This architecture page covers the practical integration path: fetching a feed, validating its structure, normalizing indicators, and loading them into your analysis environment. It matters whenever your internal telemetry needs enrichment with up‑to‑date adversary intelligence, whether you’re building a SIEM correlation rule, a firewall block list, or a threat‑hunting query.

Fetching the Feed

The most common entry point is an HTTP endpoint that delivers a JSON, STIX, or TAXII payload. The canonical pattern uses curl with strict TLS and timeout flags to prevent hanging or silent downgrades.

curl -sSf \
  --tlsv1.3 \
  --connect-timeout 10 \
  --max-time 30 \
  -H "Authorization: Bearer $TI_API_TOKEN" \
  -H "Accept: application/json" \
  -o /var/lib/threat-intel/raw_feed.json \
  "https://feed.example.org/indicators"

• --sSf combines --show-error, --silent, and --fail. If the server returns a 4xx or 5xx, curl exits with a non‑zero status and prints the HTTP status line.
• --tlsv1.3 forces the highest supported protocol. Omitting this flag allows the stack to fall back to TLS 1.2, which may be acceptable but exposes you to older cipher suites.
• --connect-timeout and --max-time bound the network window. Without them, a unresponsive feed can block your scheduler indefinitely.

If the feed uses API‑key authentication instead of a bearer token, the header becomes -H "X-API-Key: $TI_API_KEY". Some providers embed the key as a query parameter (?api_key=…), but passing it in a header is preferred to avoid leakage in server logs or browser history.

Common failure mode

A frequent silent failure occurs when the feed redirects (HTTP 302/301) and curl follows the redirect but the final destination presents a certificate signed by an unknown CA. The --fail flag only checks the initial response code; the redirected request may succeed with an invalid cert, and curl will surface the TLS error only if --proto = =https is not restricted or if --proxy-insecure is not set. The safe pattern is:

curl -sSf --tlsv1.3 --proto = "=https" --tlsv1.3 --cacert /path/to/ca-bundle.crt \
  -H "Authorization: Bearer $TI_API_TOKEN" \
  -o /var/lib/threat-intel/raw_feed.json \
  "https://feed.example.org/indicators"

--cacert pins the trusted CA bundle. If the feed’s certificate chain is not rooted in the bundle supplied, curl aborts with schannel: failed to verify certificate's CN (on Windows) or SSL: certificate verify failed (on Unix), and the pipeline stops before data is written.

Validating Format and Schema

Once the raw payload is on disk, the next step is confirming that the structure matches what your downstream consumers expect. Feeds often advertise a format (STIX 2.1, OpenIOC, CSV, custom JSON) but deviate in practice.

A lightweight validation step can be written in Python using the standard library plus jsonschema, or in jq for pure JSON streams.

Python example (generic, no product‑specific libraries):

#!/usr/bin/env python3
import json, sys, jsonschema
from pathlib import Path

schema = json.loads(Path("ti_feed_schema.json").read_text())
feed   = json.loads(Path("/var/lib/threat-intel/raw_feed.json").read_text())

try:
    jsonschema.validate(feed, schema)
    print("Schema validation passed")
except jsonschema.ValidationError as err:
    print(f"Schema error: {err.message}", file=sys.stderr)
    sys.exit(1)

The schema file (ti_feed_schema.json) is the single source of truth for what your integration accepts. Typical keys include type, spec_version, indicators, generated_at, and confidence scores. If the feed uses STIX, the schema will reference the STIX JSON schema package; if it’s a custom provider, you define the minimal set of fields required for your enrichment pipeline.

jq one‑liner for quick checks:

jq 'has("indicators") and (.indicators | length > 0)' /var/lib/threat-intel/raw_feed.json

This exits with status 0 if the indicators array exists and is non‑empty, otherwise non‑zero. It’s useful in shell pipelines to branch between “process the feed” and “log and skip”.

Sharp edge: format mis‑labeling

A feed may advertise application/stix+json; charset=utf-8 while actually delivering plain JSON with a top‑level objects key instead of indicators. Conversely, some providers wrap their payload in a { "feed": { ... } } envelope. The validation step must be tolerant of both layouts or normalize the envelope before schema validation. A common pattern is a small normalization function that extracts the inner payload regardless of wrapper name, then validates the extracted structure.

Normalizing Indicators

Raw feed data is rarely in the exact shape required by your internal models. Normalization translates observed attributes into a canonical form your tools understand.

Python normalization sketch (generic):

import datetime

def normalize(indicator):
    # Map common field names to canonical names
    mapping = {
        "title": "title",
        "description": "description",
        "pattern": "pattern",
        "pattern_type": "pattern_type",
        "confidence": "confidence",
        "first_seen": "first_seen",
        "last_seen": "last_seen",
        "source_name": "source_name",
        "labels": "labels",
    }

    out = {}
    for src, dst in mapping.items():
        val = indicator.get(src)
        if val is not None:
            out[dst] = val

    # Coerce timestamps to ISO‑8601 strings
    for ts_key in ("first_seen", "last_seen"):
        if ts_key in out and isinstance(out[ts_key], (int, float)):
            dt = datetime.datetime.utcfromtimestamp(out[ts_key] / 1000)
            out[ts_key] = dt.isoformat() + "Z"

    # Ensure confidence is an integer 0‑100
    if "confidence" in out:
        try:
            out["confidence"] = int(out["confidence"])
        except (ValueError, TypeError):
            out["confidence"] = 50  # default midpoint

    return out

normalized = [normalize(i) for i in feed.get("indicators", [])]

The function above is deliberately primitive—it does not implement full STIX pattern parsing or observable type resolution. Those are valid downstream steps, but they belong to the consumption layer, not the integration pipeline. The key takeaway is that normalization should be idempotent and tolerant of missing or extra fields.

Sharp edge: confidence score semantics

Different feeds use different confidence scales. Some express it as a percentage (0‑100), others as a severity enum (low/medium/high), and some as a numeric score tied to a proprietary reputation system. If your pipeline assumes a 0‑100 integer and the incoming feed supplies "confidence": "high", the downstream risk scorer may treat it as 0 or crash. Always map the incoming confidence value to your internal scale at ingestion time, and log the original value alongside the normalized one for auditability.

Loading into a Security Data Store

After normalization, indicators must be persisted where analysts and automation can query them. The choice of store depends on your existing stack, but the integration mechanism shares common patterns.

PostgreSQL COPY example:

psql -h db.example -U security_user -d threat_intel -c \
  "\copy indicators(observable, type, confidence, first_seen, last_seen, source) \
   FROM '/var/lib/threat-intel/normalized.csv' \
   WITH (FORMAT csv, HEADER true, DELIMITER ',')"

The CSV should have one row per indicator, with columns matching the schema above. COPY is bulk‑loading efficient and avoids the overhead of individual INSERT statements. If your store is Elasticsearch, the equivalent is the bulk API:

curl -sSf -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ES_TOKEN" \
  -X POST \
  -H "Accept: application/json" \
  "https://es.example.org:9200/threat-intel/_bulk" \
  -d @/var/lib/threat-intel/bulk_payload.json

The bulk file format alternates between an action line and a JSON document line:

{"index":{"_index":"threat-intel","_id":"12345"}}
{"observable":"1.2.3.4","type":"ipv4-addr","confidence":85,"first_seen":"2023-07-12T00:00:00Z",...}

Sharp edge: deduplication and stampedes

When a feed is re‑fetched on a schedule, the same indicators may arrive multiple times. Without idempotent writes, you’ll accumulate duplicates, inflating alert volumes and potentially triggering false‑positive blocks. The safest pattern is to enforce a unique constraint on the combination of observable + type + source_name, and use an “upsert” (INSERT … ON CONFLICT DO UPDATE) pattern. If your store does not support upserts natively, a pre‑flight SELECT for the key tuple before each write is required. Forgetting this step is a common production incident: a daily feed run silently adds thousands of duplicate rows, and subsequent SIEM rules start firing on previously‑seen noise.

Scheduling and Automation

Integration is useless if it doesn’t run regularly. The deployment pattern mirrors any cron‑ or systemd‑driven job, with extra care for error handling and alerting.

systemd service and timer (generic unit files):

/etc/systemd/system/ti-feed-loader.service:

[Unit]
Description=Threat Intelligence Feed Loader
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ti-feed-loader.sh
Restart=on-failure
RestartSec=30

/etc/systemd/system/ti-feed-loader.timer:

[Unit]
Description=Run TI feed loader every 4 hours

[Timer]
OnBootSec=15min
OnUnitActiveSec=4h
Persistent=true

[Install]
WantedBy=timers.target

/usr/local/bin/ti-feed-loader.sh:

#!/usr/bin/env bash
set -euo pipefail

LOG="/var/log/ti-feed-loader.log"
RAW="/var/lib/threat-intel/raw_feed.json"
NORM="/var/lib/threat-intel/normalized.csv"

# 1. Fetch
curl -sSf --tlsv1.3 --connect-timeout 10 --max-time 30 \
  -H "Authorization: Bearer $TI_API_TOKEN" \
  -o "$RAW" "https://feed.example.org/indicators" >>"$LOG" 2>&1 || {
  echo "$(date --iso-8601=seconds) FAIL: curl fetch exited $?" >>"$LOG"
  exit 1
}

# 2. Validate schema
python3 /usr/local/bin/validate_schema.py "$RAW" >>"$LOG" 2>&1 || {
  echo "$(date --iso-8601=seconds) FAIL: schema validation failed" >>"$LOG"
  exit 1
}

# 3. Normalize
python3 /usr/local/bin/normalize.py "$RAW" > "$NORM" >>"$LOG" 2>&1 || {
  echo "$(date --iso-8601=seconds) FAIL: normalization error" >>"$LOG"
  exit 1
}

# 4. Load into store
psql -h db.example -U security_user -d threat_intel \
  -c "\copy indicators(...) FROM '$NORM' WITH (FORMAT csv, HEADER true)" >>"$LOG" 2>&1 || {
  echo "$(date --iso-8601=seconds) FAIL: db load error" >>"$LOG"
  exit 1
}

# 5. Success hook (optional)
systemctl --quiet reload ti-rules.service

Make the script executable: chmod +x /usr/local/bin/ti-feed-loader.timer. Enable and start the timer system-wide:

systemctl daemon-reload
systemctl enable --now ti-feed-loader.timer

The Persistent=true flag ensures that if the host was offline when the timer was supposed to fire, the job runs immediately on next boot.

Sharp edge: timer drift and overlapping runs

If a feed takes longer than the interval to process (e.g., a large STIX bundle with deep schema validation), the next timer tick may start before the previous run finishes. The set -euo pipefail and explicit exit 1 on failure will cause systemd to record a failure, but the next scheduled run may still launch. A guard file or flock can prevent overlap:

exec 9>/var/lock/ti-feed-loader.lock
flock -n 9 || { echo "$(date --iso-8601=seconds) Another instance is running" >>"$LOG"; exit 0; }

The lock is acquired at script start and released on exit. If another instance holds the lock, the new run logs a message and exits cleanly, preventing concurrent database writes and duplicate indicator ingestion.

Sharp Edges — What Silently Fails, What You Learn Too Late

Symptom	Root Cause	Discovery Path
curl exits 0 but raw_feed.json is empty or contains HTML error page	Feed redirects to a captcha/portal page; curl follows the redirect but the final HTML is not JSON	Inspect curl -v output for Location: header and final HTTP status
Schema validation passes, but downstream enrichment produces KeyError: 'pattern'	Feed uses a non‑standard pattern field name (e.g., pattern_text or criteria)	Add a field‑mapping layer before validation; log raw keys for inspection
Indicators appear in the DB with first_seen timestamps 5 years in the past	Feed supplies Unix epoch in milliseconds, normalization divides by 1000 but source uses seconds, or timezone offset is ignored	Compare first_seen values against known campaign start dates; add unit‑detection logic
psql COPY fails with invalid byte sequence for encoding "UTF‑8"	Feed CSV contains Windows‑style \r\n line endings or non‑UTF‑8 characters	Run file on the CSV; convert with dos2unicode or adjust psql client encoding
Daily feed run succeeds but no new indicators appear	Deduplication key (observable + type + source) matches all existing rows; upsert silently skips writes	Query the store for recent counts; add a “last_seen” column and update on each ingest to detect “no‑new‑data” cycles
TLS error after OS update: ssl_module.c:1234: EOF	System library upgraded, curl now negotiates a protocol version the feed endpoint no longer supports	Check curl --version; adjust --tlsv1.2 or pin a specific version if the feed has not modernized

Checklist before promoting a new feed integration to production

1. Fetch test – Run curl manually against the endpoint; verify the exit code, TLS version, and that the output file contains valid JSON or CSV.
2. Schema test – Run the validation script on a sample payload; confirm that both expected and edge‑case structures are handled.
3. Normalization test – Insert a known‑good indicator and a marginal one (missing fields, unusual confidence format) into a throwaway DB table; verify the normalization script produces the expected output without exceptions.
4. Load test – Use psql COPY or the bulk API with a batch of 1 000+ rows; confirm throughput and that duplicate‑key handling works.
5. Timer test – Enable the systemd timer; let it run for two full cycles; check the log for any FAIL entries, successful loads, and lock‑contention messages.
6. Rollback plan – Keep the previous feed URL, schema, and loader script pinned in version control. If the new feed causes a production incident, revert the service unit and timer to the last known‑good commit.

Failure‑Mode Summary

• Network‑level: DNS failures, TLS certificate mismatches, firewall blocks, or unexpected HTTP redirects can cause curl to abort or write non‑payload data. Always pair --fail with --cacert and bounded timeouts.
• Schema‑level: Feeds may change field names, drop required keys, or alter spec_version. A strict validator will halt the pipeline; a permissive one will push bad data downstream. The sweet spot is a validation step that *rejects* missing mandatory fields but *coerces* optional ones with defaults.
• Data‑level: Confidence scales, timestamp units, and observable formats vary across providers. Normalization must be explicit about conversion rules and log the original values for post‑mortem analysis.
• Store‑level: Duplicate primary keys, encoding mismatches, and missing upsert semantics are the most common causes of silent data loss or bloat. Enforce uniqueness at the database level and use idempotent write paths.
• Scheduling‑level: Overlapping timer runs, lock contention, and silent failures (non‑zero exit codes ignored by the scheduler) turn a reliable daily enrichment into a source of entropy. Use flock or Persistent=true to serialize execution.

The architecture described here is not tied to any commercial platform’s API, UI, or configuration language. It maps the generic HTTP‑fetch → validate → normalize → load pipeline that underlies most threat intelligence ingestion workflows, and the concrete commands, flags, and file‑format patterns shown are drawn from publicly available tools (curl, jq, psql, systemd, python3). Readers can replace the example endpoint URL, authentication header, or database connection details with the specifics of their own environment while retaining the same structural guarantees and failure‑avoidance patterns.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.