Hybrid Cloud Networking | The Garnet Wiki

Hybrid cloud networking connects on-premises infrastructure with one or more public cloud providers, enabling workloads to span both environments seamlessly. It matters when applications must maintain low-latency access to local data, scale dynamically using cloud resources, or meet regulatory requirements for data residency. This is not simply about moving data between cloud and data center—it is about creating a unified network fabric where services can discover, communicate, and scale across boundaries with consistent performance, security, and operational simplicity.

Establishing Connectivity

To connect a data center to a public cloud, the foundational step is establishing a reliable network path. This requires configuring a virtual private network (VPN) or a dedicated private link.

For site-to-site VPNs, the process begins with creating a virtual network gateway in the cloud provider. The gateway exposes a public IP address and supports IPsec tunnels. On-premises, a router or firewall (e.g., Cisco ASA, Palo Alto, or Linux strongswan) must initiate IPsec sessions to the cloud gateway.


ipsec.conf
conn cloud-connection
    left=192.168.1.10
    [email protected]
    leftsubnet=10.0.0.0/8
    right=203.0.113.5
    rightid=203.0.113.5
    rightsubnet=10.1.0.0/16
    ike=aes256-sha256-modp2048
    esp=aes256-sha256
    auto=start

The auto=start directive ensures the tunnel is established at system boot and reconnected after failures. The rightsubnet defines which cloud subnets are reachable via this tunnel. The cloud provider provides the right IP and rightid, which may be a DNS name or IP address.

A common failure mode is mismatched ike and esp proposals: the cloud gateway may expect aes256-sha256-modp1024, but the on-premises router uses modp2048. The tunnel fails silently—no logs, no alerts—until latency spikes or service timeouts occur. The solution is to verify the ike and esp values using ipsec status and ipsec auto --status.

For dedicated private links, use a service like AWS Direct Connect, Azure ExpressRoute, or Google Cloud Interconnect. These provide a direct physical connection, typically via a cross-connect at a colocation facility.

# Example: Creating a Direct Connect connection via AWS CLI
aws directconnect create-connection \
    --location "Eq1" \
    --bandwidth "1Gbps" \
    --connection-name "onprem-to-aws" \
    --vlan 101 \
    --provider-name "Equinix" \
    --tags Key=Environment,Value=Production

The vlan value must match the VLAN configuration on the customer edge (CE) device. If the CE device is misconfigured and assigns a different VLAN than the cloud side, packets traverse the link but fail to reach the intended VPC.

Routing Across Environments

Once connectivity is established, traffic must flow between on-premises and cloud networks. This requires consistent and dynamic routing.

Cloud providers support route propagation from virtual networks to gateways. A route table in the cloud defines how packets are forwarded. On-premises, routers use static routes or dynamic protocols.

# Example: Adding a static route on an on-premises router (Linux with Quagga)
# /etc/quagga/zebra.conf
hostname r1
log file /var/log/zebra.log
interface eth0
 ip address 192.168.1.10/24
!
router bgp 65000
 bgp router-id 192.168.1.10
 network 10.0.0.0/8
 neighbor 203.0.113.5 remote-as 65001
 neighbor 203.0.113.5 update-source eth0
!

The network 10.0.0.0/8 command advertises the on-premises network to the cloud. The cloud BGP speaker (e.g., AWS Route53, Azure BGP, or a cloud router) receives the advertisement and populates its routing table.

A subtle failure occurs when the cloud side advertises a route but the on-premises router does not accept it. The cloud provider may set a local-preference value, but the on-premises BGP router does not import the route into the routing table. The fix is to configure import and export policies:

# On-premises BGP: import routes from cloud
router bgp 65000
 neighbor 203.0.113.5 route-map IMPORT-CLOUD in
!
route-map IMPORT-CLOUD permit 10
 match community CLOUD-ROUTES
 set local-preference 200

The community attribute is crucial: the cloud provider tags routes with CLOUD-ROUTES, and the on-premises router imports them with a higher local preference, ensuring cloud traffic takes the preferred path.

Service Discovery and Load Balancing

For applications to span on-premises and cloud, services must be discoverable across the network. This is achieved through DNS and service discovery.

A common pattern is to register services in a shared DNS zone. On-premises services publish A records, cloud services publish CNAMEs or SRV records.

# Example: Registering a cloud service via cloud provider API
curl -X POST https://api.example.com/v1/dns/records \
  -H "Content-Type: application/json" \
  -d '{
    "name": "api.production.example.com",
    "type": "CNAME",
    "value": "myapp-1234567890.elb.us-east-1.amazonaws.com",
    "ttl": 300
  }'

The ttl value determines how long resolvers cache the record. A low TTL (e.g., 60) improves failover responsiveness but increases DNS query load. A high TTL (e.g., 3600) reduces load but delays service changes.

When a service is available in both environments, load balancing must be coordinated across them. Cloud load balancers (e.g., AWS ELB, Azure Application Gateway) can route traffic to on-premises endpoints via a virtual IP (VIP) or by adding on-premises instances to a pool.

# Example: Adding an on-premises instance to a cloud load balancer
aws elb register-instances-with-load-balancer \
  --load-balancer-name "web-lb" \
  --instances i-12345678 i-87654321 \
  --regions us-east-1 us-west-2

The regions field allows cross-region load balancing, but the load balancer must be able to reach on-premises instances via a public IP or a private link. If the on-premises instance is behind a NAT, the health check may fail: the load balancer sends a request to 10.0.1.10, but the NAT translates it to 192.168.1.10. The instance replies, but the load balancer receives the packet at the public IP, not the internal one.

The solution is to configure the load balancer to use the X-Forwarded-For header for health checks and to set the source-dst-port mapping in the NAT.

Security and Identity Across Boundaries

Security is the glue that makes hybrid networking reliable. Access control, firewalls, and identity must span cloud and data center.

A common failure is misconfigured security groups or network ACLs that allow traffic in one direction but not the other.

# Example: Defining a security group rule for cloud-to-on-premises traffic
aws ec2 authorize-security-group-ingress \
  --group-id sg-12345678 \
  --protocol tcp \
  --port 80 \
  --source-group sg-98765432 \
  --description "Allow HTTP from on-premises"

This rule allows traffic from the on-premises security group (sg-98765432) to the cloud instance group (sg-12345678). The cloud instance group must be defined in the same VPC and include the on-premises VPC as a peer.

The critical edge case is when on-premises instances are not in the same VPC as the cloud instances. The solution is to create a VPC peering connection or use AWS Transit Gateway, Azure Virtual WAN, or a custom routing mesh.

# Example: Creating a VPC peering connection and enabling routing
aws ec2 create-vpc-peering-connection \
  --vpc-id vpc-12345678 \
  --peer-vpc-id vpc-87654321 \
  --peer-region us-west-2 \
  --tag-specifications '[
    {
      "ResourceType": "vpc-peering-connection",
      "Tags": [
        {
          "Key": "Name",
          "Value": "onprem-to-cloud"
        }
      ]
    }
  ]'

After creation, the peering connection must be accepted by the peer VPC owner. The connection is active but traffic may still fail due to missing route table entries.

# Example: Adding a route in a cloud route table to peer VPC
aws ec2 create-route \
  --route-table-id rtb-12345678 \
  --destination-cidr-block 10.0.0.0/16 \
  --vpc-peering-connection-id pcx-12345678

The destination-cidr-block must exactly match the on-premises network. A common error is using 10.0.0.0/8 instead of 10.0.0.0/16, causing traffic to be routed to the wrong VPC.

Identity synchronization is another challenge. Users and services must be authenticated across environments. A common pattern is to use an identity provider (IdP) such as Keycloak, Okta, or Azure AD.

# Example: Configuring SAML federation between on-premises and cloud
curl -X POST https://idp.example.com/api/saml/metadata \
  -H "Content-Type: application/json" \
  -d '{
    "sp": {
      "entityId": "https://app.cloud.example.com",
      "acsUrl": "https://app.cloud.example.com/sso/acs",
      "nameIdFormat": "urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"
    },
    "idp": {
      "entityId": "https://idp.example.com",
      "ssoUrl": "https://idp.example.com/sso",
      "x509Cert": "-----BEGIN CERTIFICATE-----\nMIID...\n-----END CERTIFICATE-----"
    }
  }'

The acsUrl is the endpoint where the cloud application receives the SAML response. The nameIdFormat determines how the IdP identifies users. A mismatch between nameIdFormat and the expected format in the cloud app causes login failures.

Monitoring and Troubleshooting

Hybrid networking is only as reliable as its observability. Monitoring must span both environments.

Key metrics include:

Tunnel uptime (IPsec, BGP, MPLS)
Latency between on-premises and cloud
Packet loss across the network
Load balancer health check status
DNS resolution time

The most common failure is a missing health check: the load balancer reports a cloud instance as healthy, but the instance is unreachable from on-premises. The fix is to configure the health check to probe from multiple locations.

# Example: Configuring a multi-location health check for a cloud service
curl -X PUT https://api.example.com/v1/healthchecks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "web-service",
    "type": "HTTP",
    "interval": "30s",
    "timeout": "10s",
    "url": "http://api.example.com/health",
    "locations": [
      "us-east-1",
      "us-west-2",
      "eu-west-1"
    ],
    "expectedStatus": 200,
    "expectedBody": "OK"
  }'

The locations field ensures that health checks originate from each region, catching issues like regional outages or route failures.

When a service fails, the first step is to verify the network path.

# Example: Tracing a packet from on-premises to cloud service
traceroute -n -m 30 203.0.113.10

A missing hop between on-premises and cloud indicates a routing or firewall issue. The next step is to check the BGP session.

# Example: Inspecting BGP session state
show ip bgp summary

Output includes:

Neighbor, State, Up/Down, Received, Sent, Last reset
A State of Idle or Active suggests a BGP session failure.

A Last reset time that matches a reboot or network outage points to a root cause.

Finally, packet capture is essential. Use tcpdump on both on-premises and cloud instances to capture traffic during a failure.

# Example: Capturing traffic on a cloud instance
tcpdump -i eth0 -n -s 0 -w /var/capture.pcap host 10.0.1.10

The host 10.0.1.10 filter captures only traffic to the on-premises service. Analyze the file with wireshark or tshark to identify missing TCP handshakes, failed TLS handshakes, or incorrect routing.

Resilience and Failover

In a hybrid setup, failover must be automatic and predictable. The most common failure is a delay in failover: the primary cloud service becomes unavailable, but the on-premises service takes minutes to respond.

The solution is to use a combination of health checks, DNS failover, and routing protocols.

# Example: Configuring DNS-based failover with TTL and health checks
curl -X PUT https://api.example.com/v1/dns/records \
  -H "Content-Type: application/json" \
  -d '{
    "name": "app.example.com",
    "type": "A",
    "values": [
      "192.0.2.10",
      "192.0.2.11"
    ],
    "ttl": 60,
    "healthCheck": "health-check-123"
  }'

The healthCheck field links the DNS record to a health check. If the primary IP (192.0.2.10) fails, the DNS server updates the record to prioritize the secondary (192.0.2.11). The ttl value determines how long clients cache the record.

Failover can be enhanced by using a cloud-based orchestration tool like Kubernetes or a service mesh (Istio, Linkerd). These tools can dynamically route traffic based on service health, latency, and load.

# Example: Kubernetes Service with cloud and on-premises endpoints
apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080
  externalTrafficPolicy: Local
  loadBalancerIPs:
    - 192.0.2.100
    - 192.0.2.101

The externalTrafficPolicy: Local ensures that traffic is routed directly to the node where the pod is running, minimizing latency. The loadBalancerIPs field allows the service to be accessible from both cloud and on-premises networks.

Failover is not just about IP addresses—it is about state. When a cloud service fails, on-premises services must maintain session state, often through shared databases or caching layers.

The ultimate test is a network partition: the cloud becomes unreachable, and the on-premises network must continue serving users. The success of this test depends on correct routing, DNS, and identity synchronization.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.