AWS Networking & Connectivity (VPCs, Transit Gateway, Peering, Hybrid)

AWS networking can feel daunting—CIDRs, route tables, NAT gateways, security groups, NACLs, VPC peering, AWS Transit Gateway, AWS Direct Connect, and hybrid patterns all collide. This deep-dive gives you a clear blueprint for multi-account design, a precise troubleshooting runbook for connectivity issues, and a cost/latency checklist you can apply immediately.

1) Core Building Blocks (Quick Refresh)

2) Multi-Account, Multi-VPC Architecture: Peering vs. Transit Gateway

Many teams start with VPC peering for a few accounts, then hit a wall when connectivity grows. AWS Transit Gateway becomes the central hub for hundreds of attachments, route domains (Routable & Blackhole routes), and inter-region peering between TGWs. Use these decision cues:

Reference Patterns

• Spoke VPCs per workload Shared Services VPC Inspection VPC via GWLB or NVA.
• Segment routes with multiple TGW route tables: prod, non-prod, shared, inspection.
• Inter-Region: TGW-to-TGW peering + selective route propagation.

3) Step-by-Step Troubleshooting: VPC Connectivity (Routing, NAT, Peering, SG/NACL)

Use this structured runbook to reduce MTTR. Move top-to-bottom without skipping.

CLI/Scripted Checks (AWS Tools for PowerShell)

Install module: Install-Module -Name AWSPowerShell.NetCore -Scope CurrentUser

# List VPCs, CIDRs, and DNS settings
Get-EC2Vpc | Select-Object VpcId, CidrBlock, IsDefault, EnableDnsSupport, EnableDnsHostnames

# Show subnets + route table association
Get-EC2Subnet | Select-Object SubnetId, VpcId, CidrBlock, AvailabilityZone, RouteTableAssociationId

# Enumerate route tables and routes (spot black-holes)
Get-EC2RouteTable | ForEach-Object {
  $_.Routes | ForEach-Object {
    [PSCustomObject]@{
      RouteTableId = $_.RouteTableId
      Destination  = $_.DestinationCidrBlock
      State        = $_.State
      Target       = ($_.GatewayId,$_.NatGatewayId,$_.TransitGatewayId,$_.VpcPeeringConnectionId,$_.NetworkInterfaceId) -join '/'
    }
  }
} | Sort-Object RouteTableId, Destination | Format-Table -AutoSize

# Check peering connections and status
Get-EC2VpcPeeringConnection | Select-Object VpcPeeringConnectionId, Status, RequesterVpcInfo, AccepterVpcInfo

# Check Transit Gateway attachments and route tables
Get-EC2TransitGatewayAttachment | Select-Object TransitGatewayAttachmentId, TransitGatewayId, ResourceType, ResourceId, State

Get-EC2TransitGatewayRouteTable | ForEach-Object {
  $rt = $_
  Get-EC2TransitGatewayRoute -TransitGatewayRouteTableId $rt.TransitGatewayRouteTableId | ForEach-Object {
    [PSCustomObject]@{
      TgwRouteTableId = $rt.TransitGatewayRouteTableId
      Destination     = $_.DestinationCidrBlock
      Type            = $_.Type
      State           = $_.State
      Attachment      = ($_.TransitGatewayAttachments | Select-Object -ExpandProperty TransitGatewayAttachmentId) -join ','
    }
  }
} | Format-Table -AutoSize

# VPN tunnels health (if using Virtual Private Gateways)
Get-EC2VpnConnection | Select-Object VpnConnectionId, State, CustomerGatewayConfiguration, VpnGatewayId, TransitGatewayId

Interpretation tips:

• Peering: Status must be active on both sides; ensure routes in both VPCs include each other’s CIDRs via the peering ID.
• TGW: Attachments should be available. The spoke VPC subnets must associate with the correct VPC route tables that point to the TGW.
• Black-hole: Route state shows blackhole when target is missing or down—fix target or delete/replace route.

4) Hybrid Connectivity: On-Prem to AWS (VPN vs. Direct Connect)

For migrations, the common path starts with Site-to-Site VPN (fast, internet-based), then moves to Direct Connect for predictable latency and cost. Many enterprises run both: DX for primary, VPN for backup.

Design Notes

• Redundancy: two VPN tunnels per connection; for DX, use Link Aggregation Group (LAG) or dual circuits across different providers/POPs.
• Routing: Use BGP for failover and route summarization; watch for longer-prefix wins causing unintended path selection.
• MTU: Set consistent MTU across routers and EC2; enable jumbo frames where supported end-to-end.

# List TGW route tables and identify learned (propagated) routes from on-prem
$tgwRts = Get-EC2TransitGatewayRouteTable
foreach($t in $tgwRts){
  Write-Host "== TGW RT:" $t.TransitGatewayRouteTableId
  Get-EC2TransitGatewayRoute -TransitGatewayRouteTableId $t.TransitGatewayRouteTableId |
    Where-Object { $_.Type -eq 'propagated' } |
    Select-Object DestinationCidrBlock, State, @( @{n='Attachments'; e={ ($_.TransitGatewayAttachments | % TransitGatewayAttachmentId) -join ',' } } ) |
    Format-Table -AutoSize
}

5) Data Transfer, Egress Fees & Latency-Optimized Design

Networking cost is often invisible until the bill arrives. Keep traffic in-Region, in-AZ where possible, and avoid unnecessary hairpins through inspection VPCs. When you must cross Regions or the public internet, pick the right connector and placement.

6) End-to-End Runbook: “It works in VPC-A, but not from VPC-B”

1. Identify path: VPC-B → TGW → VPC-A (App). Get source/destination IP:port.
2. Check SG/NACL: Allow from VPC-B CIDR to target port. NACL allow both directions.
3. Verify TGW association: VPC-B attachment associated to correct TGW route table.
4. Check TGW routes: Is VPC-A CIDR present (static or propagated)? State = active.
5. Spoke route tables: Subnets in VPC-B have a route to VPC-A CIDR via TGW; VPC-A subnets route back to VPC-B via TGW.
6. DNS & MTU: Resolve to private IP; test with ping -s/ping -l (size) to detect fragmentation issues.
7. Re-test with bypass: Temporarily bypass inspection/firewall to isolate whether middlebox is the culprit.

# Requires EC2 Instance Connect or SSM Session Manager on the source host
# This uses SSM to run reachability checks without public SSH/RDP

$instanceId = "i-xxxxxxxxxxxx"
$dest = "10.1.20.15"
$port = 5432

Send-SSMCommand -InstanceId $instanceId -DocumentName "AWS-RunShellScript" -Parameter @{ commands = @(
  "date",
  "echo Testing to $dest:$port",
  "timeout 3 bash -c '

resource "aws_ec2_transit_gateway" "hub" {
  description = "Org TGW"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
  dns_support  = "enable"
  vpn_ecmp_support = "enable"
}

resource "aws_ec2_transit_gateway_route_table" "prod" {
  transit_gateway_id = aws_ec2_transit_gateway.hub.id
}

# Associate VPC attachments to prod table; add static routes if needed

{
  "Resources": {
    "S3GatewayEndpoint": {
      "Type": "AWS::EC2::VPCEndpoint",
      "Properties": {
        "ServiceName": { "Fn::Sub": "com.amazonaws.${AWS::Region}.s3" },
        "VpcId": { "Ref": "VPC" },
        "RouteTableIds": [{ "Ref": "PrivateRtA" }, { "Ref": "PrivateRtB" }],
        "VpcEndpointType": "Gateway"
      }
    }
  }
}

# Find drops from a specific source to a destination in VPC Flow Logs (Athena)
SELECT * FROM vpc_flow_logs
WHERE srcaddr = '10.0.12.34'
  AND dstaddr = '10.1.20.15'
  AND action = 'REJECT'
  AND start >= to_unixtime(current_timestamp - interval '1' hour);

# Identify subnets without explicit routes to TGW or IGW (potential isolation)
Get-EC2RouteTable | % {
  $rt = $_
  $hasTgwOrIgw = $rt.Routes | Where-Object { $_.GatewayId -like 'igw-*' -or $_.TransitGatewayId -like 'tgw-*' }
  if(-not $hasTgwOrIgw){ "Isolated? $($rt.RouteTableId)" }
}

# Find overlapping CIDRs between VPCs quickly
$vpcs = Get-EC2Vpc | Select-Object VpcId, CidrBlock
foreach($a in $vpcs){ foreach($b in $vpcs){
  if($a.VpcId -ne $b.VpcId){
    # naive check; for production use a proper CIDR overlap function
    if($a.CidrBlock.Split('/')[0] -eq $b.CidrBlock.Split('/')[0]){
      "Possible overlap pattern: $($a.VpcId) and $($b.VpcId) ($($a.CidrBlock) vs $($b.CidrBlock))"
    }
  }
}}

# Report VPN connections that are not 'available'
Get-EC2VpnConnection | Where-Object { $_.State -ne 'available' } | 
  Select-Object VpnConnectionId, State, TransitGatewayId, VpnGatewayId