ESC
Type to search guides, tutorials, and reference documentation.

Cloud Network Engineering

Cloud networking from first principles: address planning, routing and egress, connecting networks without transitivity, DNS, stateful versus stateless filtering, and the failure modes that are hardest to diagnose.

Cloud network engineering is the discipline of designing, deploying, and operating reliable, performant, and secure network infrastructure in cloud environments. It matters when applications must scale across regions, when latency and availability are critical, when traffic patterns shift unpredictably, and when failures must be detected and resolved without human intervention. Whether building a microservices platform, a real-time analytics pipeline, or a globally distributed web application, the network is the foundation upon which all other systems depend.

Core Network Constructs

Virtual Private Clouds (VPCs)

A VPC is the logical backbone of cloud networking. It provides isolated, configurable network segments with custom IP ranges, routing tables, and gateway configurations. In a typical setup, a VPC spans multiple availability zones and includes public and private subnets.

To create a VPC with AWS-like semantics using aws ec2 create-vpc, the following command is standard:

aws ec2 create-vpc \
  --cidr-block 10.0.0.0/16 \
  --instance-tenancy default \
  --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=prod-core}]'

This produces a VPC with a 16-bit address space. Subnets are then created within the VPC:

aws ec2 create-subnet \
  --vpc-id vpc-12345678 \
  --cidr-block 10.0.1.0/24 \
  --availability-zone us-west-2a \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=web-frontend}]'

The default route table must be updated to route traffic from private subnets to the internet via an Internet Gateway (IGW):

aws ec2 create-internet-gateway \
  --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=prod-igw}]'

aws ec2 attach-internet-gateway \
  --vpc-id vpc-12345678 \
  --internet-gateway-id igw-12345678

The route table associated with the public subnet must include a route to 0.0.0.0/0 via the IGW. This route is managed via aws ec2 create-route:

aws ec2 create-route \
  --route-table-id rtb-12345678 \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id igw-12345678

Failure often occurs when a route table is not associated with a subnet. The describe-subnets command reveals the mapping:

aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=vpc-12345678"

The key failure mode is route table drift: a subnet is created but not associated with any route table, leaving it isolated. A common fix is to update the route table with aws ec2 replace-route-table-association, which replaces the current association and triggers a route propagation update.

Subnetting and Routing

Subnetting is not merely a naming convention; it defines network behavior. Public subnets have routes to the IGW, private subnets do not. To route private traffic through a NAT gateway, a NAT gateway is provisioned in a public subnet:

aws ec2 create-nat-gateway \
  --allocation-id eipalloc-12345678 \
  --subnet-id subnet-12345678

The allocation-id comes from a public Elastic IP (EIP), which must be allocated prior:

aws ec2 allocate-address --domain vpc

Once the NAT gateway is created, the route table for private subnets is updated:

aws ec2 create-route \
  --route-table-id rtb-87654321 \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id nat-12345678

A critical failure mode is NAT gateway timeout. When a NAT gateway is created but its instance is not in the same availability zone as the subnet, traffic fails silently. The NAT gateway’s AvailabilityZone field must match the subnet’s AvailabilityZone. This mismatch is often missed during automation, leading to latency spikes and dropped connections.

Security Groups and Network ACLs

Security groups (SGs) are stateful, instance-level firewalls. They define ingress and egress rules per EC2 instance, container, or load balancer.

To allow HTTP and HTTPS traffic to a web server, create a security group:

aws ec2 create-security-group \
  --group-name web-sg \
  --description "Web servers in production"

Then add rules:

aws ec2 authorize-security-group-ingress \
  --group-id sg-12345678 \
  --ip-permissions \
    IpProtocol=tcp,FromPort=80,ToPort=80,IpRanges=[{CidrIp=0.0.0.0/0}] \
    IpProtocol=tcp,FromPort=443,ToPort=443,IpRanges=[{CidrIp=0.0.0.0/0}]

A common mistake is assuming SG rules are applied immediately. In reality, the network interface (ENI) must be updated via modify-network-interface-attribute, and the instance must be restarted or the ENI reattached. A silent failure occurs when a new instance is launched with a security group but no ENI update.

Network ACLs (NACLs) are stateless, subnet-level firewalls. They define ingress and egress rules with explicit order. Unlike SGs, NACLs require explicit rules for return traffic.

To add a rule allowing SSH from a specific IP range:

aws ec2 create-network-acl-entry \
  --network-acl-id acl-12345678 \
  --rule-number 100 \
  --protocol tcp \
  --port-range FromPort=22,ToPort=22 \
  --cidr-block 203.0.113.0/24 \
  --egress false \
  --rule-action ALLOW

The rule-number is critical: NACLs process rules in order, and the first matching rule determines the action. A missing rule for return traffic (e.g., SSH responses) causes dropped packets and connection timeouts.

Load Balancing and Health Checks

Load balancers distribute traffic across multiple instances. A typical setup uses an Application Load Balancer (ALB) with target groups.

Create a target group:

aws elbv2 create-target-group \
  --name web-service \
  --protocol HTTP \
  --port 80 \
  --vpc-id vpc-12345678 \
  --health-check-protocol HTTP \
  --health-check-port 80 \
  --health-check-path /health \
  --health-check-interval-seconds 30 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 3 \
  --matcher HttpCode=200

Register instances with the target group:

aws elbv2 register-targets \
  --target-group-arn arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/web-service/1234567890123456 \
  --targets Id=i-12345678,Id=i-87654321

Health check failures are indicated in the ALB console and via describe-target-health:

aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/web-service/1234567890123456

The most common failure is misconfigured health check path. A service returns 200 but the health check expects 200-299, resulting in a "HEALTHY" target that is actually "UNHEALTHY" in the ALB. The matcher.HttpCode field must match the actual response code. A subtle issue is that the health check sends a GET request with no Host header, so the service must be configured to respond to Host: <hostname>.

DNS and Traffic Routing

DNS is used to route traffic to services. Cloud providers offer managed DNS services, such as AWS Route 53, Azure DNS, or Google Cloud DNS.

Create a hosted zone:

aws route53 create-hosted-zone \
  --name example.com. \
  --caller-reference 20240515-123456

Create an A record for a web service:

aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789012 \
  --change-batch '{
    "Changes": [
      {
        "Action": "CREATE",
        "ResourceRecordSet": {
          "Name": "www.example.com.",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [
            { "Value": "192.0.2.1" }
          ]
        }
      }
    ]
  }'

The TTL (Time to Live) controls how long DNS resolvers cache the record. A low TTL allows rapid failover, but increases query load.

For traffic routing, use weighted, latency-based, or geolocation routing policies. A latency-based routing policy selects the endpoint with the lowest latency from the user’s location.

Create a latency-based record:

{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com.",
        "Type": "A",
        "SetIdentifier": "us-west-2",
        "Region": "us-west-2",
        "TTL": 60,
        "ResourceRecords": [
          { "Value": "192.0.2.10" }
        ],
        "Weight": 100
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com.",
        "Type": "A",
        "SetIdentifier": "eu-west-1",
        "Region": "eu-west-1",
        "TTL": 60,
        "ResourceRecords": [
          { "Value": "192.0.2.20" }
        ],
        "Weight": 100
      }
    }
  ]
}

A critical failure mode is DNS propagation delay. After updating a record, clients may still resolve to old IP addresses for up to the TTL duration. This is compounded when multiple records are updated simultaneously. The dig command is essential for diagnosing DNS issues:

dig @8.8.8.8 www.example.com

The SOA (Start of Authority) record reveals the current zone version and refresh interval. A missing NS record in the zone causes the DNS hierarchy to fail to resolve the root domain.

Cross-Service Network Integration

Inter-VPC Connectivity

To enable communication between VPCs, use VPC peering or transit gateways.

VPC peering is a direct network connection between two VPCs. It requires a peering connection and route table updates.

Create a peering connection:

aws ec2 create-vpc-peering-connection \
  --vpc-id vpc-12345678 \
  --peer-vpc-id vpc-87654321 \
  --peer-region us-east-1

Accept the peering connection in the peer VPC:

aws ec2 accept-vpc-peering-connection \
  --vpc-peering-connection-id pcx-12345678

Then update route tables in both VPCs to route traffic to the peer’s CIDR blocks:

aws ec2 create-route \
  --route-table-id rtb-12345678 \
  --destination-cidr-block 10.0.0.0/16 \
  --vpc-peering-connection-id pcx-12345678

A common failure is asymmetric routing: traffic from VPC A to VPC B flows correctly, but return traffic fails because the reverse route is missing in VPC B. This leads to one-way connectivity and is difficult to debug without packet captures.

Service Discovery and Internal Load Balancing

Internal services must discover each other. Use a combination of DNS, service registry, and internal load balancers.

A service running in web-frontend subnet can resolve auth-service.internal via a private DNS zone. The internal domain is served by an internal ALB that listens on port 80.

To register a service instance with an internal DNS record, a script updates Route 53:

aws route53 change-resource-record-sets \
  --hosted-zone-id Z987654321098 \
  --change-batch '{
    "Changes": [
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "auth-service.internal.",
          "Type": "A",
          "TTL": 30,
          "ResourceRecords": [
            { "Value": "10.0.1.10" }
          ]
        }
      }
    ]
  }'

The service must also register itself with a health check, often via a POST to /register on a central service registry.

Transit Gateways and Centralized Networking

For complex topologies with many VPCs, use a transit gateway. It acts as a central hub, connecting multiple VPCs and on-premises networks.

Create a transit gateway:

aws ec2 create-transit-gateway \
  --description "Central network hub"

Attach a VPC:

aws ec2 create-transit-gateway-vpc-attachment \
  --transit-gateway-id tgw-12345678 \
  --vpc-id vpc-12345678 \
  --subnet-id subnet-12345678

The transit gateway route table must be updated to route traffic between attached VPCs and on-premises networks.

A key failure mode is route table inconsistency. When a new VPC is attached, its route table may not include routes to other VPCs via the transit gateway. The transit-gateway-route-table must be explicitly updated with create-transit-gateway-route:

aws ec2 create-transit-gateway-route \
  --destination-cidr-block 10.1.0.0/16 \
  --transit-gateway-route-table-id tgw-rtb-12345678 \
  --transit-gateway-attachment-id tgw-attach-12345678

Without this, traffic between VPCs fails silently.

Troubleshooting Patterns

Network Latency and Packet Loss

Use ping and traceroute to diagnose reachability and latency.

ping -c 10 -W 2 192.0.2.1
traceroute 192.0.2.1

A high ICMP packet loss indicates network congestion. The mtr tool combines ping and traceroute continuously:

mtr 192.0.2.1

A common failure is TCP connection timeout. The server responds to SYN but the client never receives SYN-ACK. This is often due to missing firewall rules or incorrect network ACLs blocking port 22 (SSH) or 80 (HTTP).

Connectivity from the Edge

Use curl with verbose output to diagnose HTTP connectivity:

curl -v http://api.example.com/health

The --fail, --max-time, and --retry flags are essential:

curl -f -m 10 --retry 3 -v http://api.example.com/health

A silent failure occurs when a 200 response is returned but the body is empty. The service must be configured to return a non-empty body for health checks.

Monitoring and Logging

Use systemctl to check service status and logs:

systemctl status nginx
journalctl -u nginx -f

For application-level logging, use jq to parse JSON logs:

journalctl -u app-server | jq '.timestamp, .level, .message'

A critical failure mode is log retention mismatch: logs are written but not rotated or archived. The logrotate configuration must be tuned to avoid disk exhaustion.

Summary

Cloud network engineering is not a series of isolated tasks but a continuous process of alignment, validation, and troubleshooting. The network is the glue between services, and its behavior is defined by a set of well-understood, interlocking mechanisms: VPCs, subnets, routing, security, load balancing, DNS, and service discovery. Each component has subtle failure modes that are not immediately visible but become critical under load. Mastery comes not from knowing commands, but from recognizing patterns: how to diagnose a latency spike, how to ensure a new service is reachable, how to scale traffic across regions, and how to recover from a network failure that takes down the entire system.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.