Hybrid Cloud Networking | The Garnet Wiki

Hybrid Cloud Networking connects on-premises infrastructure with one or more public cloud providers, enabling seamless data flow, workload portability, and unified management across environments. It matters when applications must span physical data centers and cloud regions—when latency-sensitive services require local compute, while bursty workloads leverage cloud elasticity, and when centralized policies must govern connectivity, security, and service discovery across distributed nodes.

Establishing Connectivity

To begin hybrid networking, first define a network fabric that spans on-premises and cloud environments. This fabric must support IP routing, dynamic addressing, and end-to-end reachability.

A typical setup uses site-to-site IPsec tunnels between on-premises routers and cloud virtual networks. The tunnel configuration is defined using standard IPsec parameters. On the on-premises side, a router (e.g., Cisco ISR, Juniper MX, or Linux strongSwan) establishes a tunnel endpoint with the cloud provider's gateway.

For example, on a Linux host using strongSwan, a configuration file /etc/ipsec.conf defines a tunnel to a cloud VPC:

conn cloud-branch
    left=192.168.1.10
    leftid=192.168.1.10
    right=10.0.1.1
    rightid=10.0.1.1
    ike=aes256-sha256-modp2048
    esp=aes256-sha256
    keyexchange=ikev2
    auto=start
    type=tunnel
    dpdaction=clear
    dpdtimeout=300
    dpdinterval=60

The right IP is the cloud VPC gateway, and left is the on-premises router. The ikev2 key exchange ensures resilient rekeying and fast failover. dpdinterval=60 and dpdtimeout=300 detect dead peers every minute, triggering reconnection if no response.

On the cloud side, the VPC gateway must accept incoming traffic and route it to internal subnets. In AWS, this is done via a VpcGatewayAttachment and RouteTable entries. For example, a route table entry in AWS CLI:

aws ec2 create-route \
    --route-table-id rtb-12345678 \
    --destination-cidr-block 192.168.0.0/16 \
    --gateway-id vgw-12345678

This route sends traffic destined for the on-premises network (192.168.0.0/16) to the virtual private gateway attached to the VPC, which is the device that carries it down the IPsec tunnel. Pointing this route at an internet gateway (igw-) instead is a common and quiet mistake: an internet gateway exists to reach the public internet, so return traffic is routed out to the internet rather than through the tunnel, and the branch office never reaches the VPC. The symptom is a tunnel that reports UP on both ends while every connection times out.

A common failure mode is missing route-table entries for subnets within the cloud VPC. Without them, services behind a NAT gateway or EC2 instance in a private subnet remain unreachable from on-premises systems. The solution is to ensure every subnet has a route to the VPC router, and that the cloud VPC gateway is properly configured to accept traffic on the tunnel’s right IP.

Inter-Service Communication Across Environments

Once connectivity is established, services must discover and communicate across boundaries. This requires consistent service discovery, load balancing, and cross-network traffic routing.

In a hybrid deployment, services often run both on-premises and in the cloud. A user-facing API might be hosted on-premises for low-latency access, while background processing runs in the cloud. The challenge is to make these services appear as a single logical network.

Use a service mesh to manage inter-service communication. Deploy a sidecar proxy (e.g., Istio with Envoy) on every service instance. The proxy handles routing, retries, circuit breaking, and metadata propagation.

A DestinationRule in Istio configures how traffic is routed between cloud and on-premises services:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: api-service
spec:
  host: api.example.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    connectionPool:
      tcp:
        maxConnections: 100
        connectTimeout: 10s
      http:
        http1MaxPendingRequests: 100
        maxRequestsPerConnection: 100
  subsets:
    - name: on-prem
      labels:
        location: onprem
    - name: cloud
      labels:
        location: cloud

This rule ensures that requests to api.example.com are load-balanced between on-premises and cloud instances, with distinct routing policies for each subset.

To route traffic between environments, use a VirtualService that defines how traffic flows through the mesh:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-virtual
spec:
  hosts:
    - api.example.com
  http:
    - route:
        - destination:
            host: api.example.com
            subset: onprem
          weight: 60
        - destination:
            host: api.example.com
            subset: cloud
          weight: 40

Traffic is sent 60% to on-prem services and 40% to cloud services. A silent failure occurs when the subset labels in the DestinationRule do not exactly match those in the service deployment. A mismatch in label keys—e.g., location: cloud vs location: cloud-vm—results in traffic being routed to the wrong subset, with no error logs or metrics.

Cross-environment traffic also requires consistent DNS resolution. Deploy a shared DNS zone (e.g., hosted in AWS Route 53 or on-premises BIND) that resolves api.example.com to both on-premises and cloud IP addresses. Use a round-robin or geolocation-based DNS strategy.

When a service in the cloud calls a service on-premises, the call may fail due to firewall rules. The destination IP in the VirtualService must be reachable from the cloud VPC, and the on-premises router must allow traffic on the service’s port (e.g., 8080) from the cloud’s CIDR block. A common mistake is forgetting to open the right IP in the cloud VPC security group.

Network Policy and Security

Security is enforced at multiple layers: network, transport, and application. Hybrid networking requires consistent policies across environments.

Define a NetworkPolicy in Kubernetes to restrict traffic between cloud and on-premises pods. For example, to allow all cloud services to access a database on-premises:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-onprem-access
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 10.0.0.0/8
      ports:
        - protocol: TCP
          port: 5432

This policy allows any IP in the cloud’s 10.0.0.0/8 range (e.g., AWS VPC CIDR) to access the postgres service on port 5432.

The policy applies to pods in the cloud cluster, but on-premises services must also be configured to allow ingress from the cloud’s CIDR block. This typically involves configuring a firewall (e.g., iptables) on the on-premises network:

iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 5432 -j ACCEPT

A subtle failure occurs when the NetworkPolicy is applied but the podSelector does not match the actual pod labels. A pod labeled app: postgres-db fails to match app: postgres, resulting in unfiltered traffic and potential security exposure.

In addition, transport-layer security (TLS) must span environments. Use mutual TLS (mTLS) to secure service-to-service calls. Configure Istio to enable mTLS between all services in the mesh:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT

This ensures every service call is authenticated using certificates and encrypted. A common failure is missing CA certificates on the on-premises side. Without proper trust configuration, cloud services cannot validate on-premises service certificates, leading to TLS handshake failures and 503 errors.

Managing Failures and Observability

Hybrid networks are prone to silent failures. Monitoring must span both environments with consistent metrics and logs.

Use a central logging pipeline with Fluentd or Vector to aggregate logs from on-premises and cloud systems. Deploy a Fluentd daemon on each host, sending logs to a centralized Elasticsearch cluster.

Example Fluentd configuration for collecting logs from Kubernetes pods:

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%N%Z
  </parse>
</source>

<match kubernetes.**>
  @type elasticsearch
  host elasticsearch.example.com
  port 9200
  logstash_format true
  logstash_prefix fluentd
  flush_interval 10s
</match>

This configuration collects logs from all Kubernetes pods and forwards them to Elasticsearch with a logstash index pattern.

For metrics, deploy Prometheus across environments. Use prometheus.yml to scrape both on-premises and cloud exporters:

scrape_configs:
  - job_name: 'onprem-services'
    static_configs:
      - targets: ['192.168.1.20:9090', '192.168.1.21:9090']
  - job_name: 'cloud-ec2'
    ec2_sd_configs:
      - region: us-east-1
        role_arn: arn:aws:iam::123456789012:role/EC2-Discovery
        refresh_interval: 30s

The ec2_sd_configs dynamically discovers all EC2 instances in us-east-1 and adds them as scrape targets. A common issue is missing role_arn in the AWS IAM role, causing the Prometheus instance to fail to discover EC2 instances. The error appears as no targets scraped in the Prometheus UI.

For distributed tracing, use OpenTelemetry. Instrument services with the OpenTelemetry SDK and send traces to a collector. A simple configuration in config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  otlp:
    endpoint: collector.tracing.example.com:4317
    timeout: 10s
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

This configuration collects traces via gRPC and exports them to a central collector. A silent failure is when the collector is reachable but the endpoint is misconfigured—e.g., collector.tracing.example.com:4317 resolves to 10.0.0.10:4317, but the collector is listening on 4317 and 4318 (due to multiple services). The result is traces being dropped or delayed.

Scaling and High Availability

To scale hybrid deployments, ensure that load balancers, databases, and message queues span both environments.

Use a global load balancer (e.g., AWS Global Accelerator, Azure Traffic Manager, or a custom solution using DNS with TTL) to route users to the nearest service instance.

For example, create a DNS record in Route 53 with a latency-based routing policy:

{
  "HostedZoneId": "Z1234567890ABC",
  "RecordSets": [
    {
      "Name": "api.example.com",
      "Type": "A",
      "SetIdentifier": "us-east-1",
      "Region": "us-east-1",
      "TTL": 60,
      "ResourceRecords": [
        "10.0.1.10"
      ],
      "Weight": 100
    },
    {
      "Name": "api.example.com",
      "Type": "A",
      "SetIdentifier": "eu-west-1",
      "Region": "eu-west-1",
      "TTL": 60,
      "ResourceRecords": [
        "10.1.1.10"
      ],
      "Weight": 100
    }
  ]
}

This routes users to the closest cloud region. A failure occurs when the Region field is misspelled (us-east-1 vs us-east-1), causing the routing policy to apply to the wrong region.

For shared databases, use a multi-region setup with read replicas in each cloud region and a central on-premises database. Use connection pooling and load balancing to distribute queries.

When a cloud region fails, the load balancer must fail over to another region. A silent issue is stale DNS caches. A user in Europe accessing api.example.com may continue using a failed us-east-1 endpoint for up to 60 seconds, even after the region is marked as unavailable.

To detect and recover from such failures, use health checks and automated failover. A health check script runs every 10 seconds:

#!/bin/bash
curl -f http://api.example.com/health || exit 1

If the script returns non-zero, the load balancer marks the region as unhealthy and redirects traffic. A common oversight is not setting interval and timeout correctly in the health check configuration, causing delayed detection of failures.

Summary

Hybrid Cloud Networking is not a single technology but a coordinated system of connectivity, service discovery, security, and observability across physical and virtual environments. It succeeds when every layer—IP routing, service mesh, network policies, monitoring, and failover—works in concert. The key to reliability is consistency: identical configurations, shared tooling, and unified observability. Failures arise not from missing features, but from subtle mismatches: label keys, endpoint names, DNS TTLs, and configuration file syntax. The result is a network that functions—until it doesn’t.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.