AWS networking can feel daunting—CIDRs, route tables, NAT gateways, security groups, NACLs, VPC peering, AWS Transit Gateway, AWS Direct Connect, and hybrid patterns all collide. This deep-dive gives you a clear blueprint for multi-account design, a precise troubleshooting runbook for connectivity issues, and a cost/latency checklist you can apply immediately.

1) Core Building Blocks (Quick Refresh)

2) Multi-Account, Multi-VPC Architecture: Peering vs. Transit Gateway

Many teams start with VPC peering for a few accounts, then hit a wall when connectivity grows. AWS Transit Gateway becomes the central hub for hundreds of attachments, route domains (Routable & Blackhole routes), and inter-region peering between TGWs. Use these decision cues:

Reference Patterns

• Spoke VPCs per workload Shared Services VPC Inspection VPC via GWLB or NVA.
• Segment routes with multiple TGW route tables: prod, non-prod, shared, inspection.
• Inter-Region: TGW-to-TGW peering + selective route propagation.

3) Step-by-Step Troubleshooting: VPC Connectivity (Routing, NAT, Peering, SG/NACL)

Use this structured runbook to reduce MTTR. Move top-to-bottom without skipping.

CLI/Scripted Checks (AWS Tools for PowerShell)

Install module: Install-Module -Name AWSPowerShell.NetCore -Scope CurrentUser

# List VPCs, CIDRs, and DNS settings
Get-EC2Vpc | Select-Object VpcId, CidrBlock, IsDefault, EnableDnsSupport, EnableDnsHostnames

# Show subnets + route table association
Get-EC2Subnet | Select-Object SubnetId, VpcId, CidrBlock, AvailabilityZone, RouteTableAssociationId

# Enumerate route tables and routes (spot black-holes)
Get-EC2RouteTable | ForEach-Object {
  $_.Routes | ForEach-Object {
    [PSCustomObject]@{
      RouteTableId = $_.RouteTableId
      Destination  = $_.DestinationCidrBlock
      State        = $_.State
      Target       = ($_.GatewayId,$_.NatGatewayId,$_.TransitGatewayId,$_.VpcPeeringConnectionId,$_.NetworkInterfaceId) -join '/'
    }
  }
} | Sort-Object RouteTableId, Destination | Format-Table -AutoSize

# Check peering connections and status
Get-EC2VpcPeeringConnection | Select-Object VpcPeeringConnectionId, Status, RequesterVpcInfo, AccepterVpcInfo

# Check Transit Gateway attachments and route tables
Get-EC2TransitGatewayAttachment | Select-Object TransitGatewayAttachmentId, TransitGatewayId, ResourceType, ResourceId, State

Get-EC2TransitGatewayRouteTable | ForEach-Object {
  $rt = $_
  Get-EC2TransitGatewayRoute -TransitGatewayRouteTableId $rt.TransitGatewayRouteTableId | ForEach-Object {
    [PSCustomObject]@{
      TgwRouteTableId = $rt.TransitGatewayRouteTableId
      Destination     = $_.DestinationCidrBlock
      Type            = $_.Type
      State           = $_.State
      Attachment      = ($_.TransitGatewayAttachments | Select-Object -ExpandProperty TransitGatewayAttachmentId) -join ','
    }
  }
} | Format-Table -AutoSize

# VPN tunnels health (if using Virtual Private Gateways)
Get-EC2VpnConnection | Select-Object VpnConnectionId, State, CustomerGatewayConfiguration, VpnGatewayId, TransitGatewayId

Interpretation tips:

• Peering: Status must be active on both sides; ensure routes in both VPCs include each other’s CIDRs via the peering ID.
• TGW: Attachments should be available. The spoke VPC subnets must associate with the correct VPC route tables that point to the TGW.
• Black-hole: Route state shows blackhole when target is missing or down—fix target or delete/replace route.

4) Hybrid Connectivity: On-Prem to AWS (VPN vs. Direct Connect)

For migrations, the common path starts with Site-to-Site VPN (fast, internet-based), then moves to Direct Connect for predictable latency and cost. Many enterprises run both: DX for primary, VPN for backup.

Design Notes

• Redundancy: two VPN tunnels per connection; for DX, use Link Aggregation Group (LAG) or dual circuits across different providers/POPs.
• Routing: Use BGP for failover and route summarization; watch for longer-prefix wins causing unintended path selection.
• MTU: Set consistent MTU across routers and EC2; enable jumbo frames where supported end-to-end.

# List TGW route tables and identify learned (propagated) routes from on-prem
$tgwRts = Get-EC2TransitGatewayRouteTable
foreach($t in $tgwRts){
  Write-Host "== TGW RT:" $t.TransitGatewayRouteTableId
  Get-EC2TransitGatewayRoute -TransitGatewayRouteTableId $t.TransitGatewayRouteTableId |
    Where-Object { $_.Type -eq 'propagated' } |
    Select-Object DestinationCidrBlock, State, @( @{n='Attachments'; e={ ($_.TransitGatewayAttachments | % TransitGatewayAttachmentId) -join ',' } } ) |
    Format-Table -AutoSize
}

5) Data Transfer, Egress Fees & Latency-Optimized Design

Networking cost is often invisible until the bill arrives. Keep traffic in-Region, in-AZ where possible, and avoid unnecessary hairpins through inspection VPCs. When you must cross Regions or the public internet, pick the right connector and placement.

6) End-to-End Runbook: “It works in VPC-A, but not from VPC-B”

1. Identify path: VPC-B → TGW → VPC-A (App). Get source/destination IP:port.
2. Check SG/NACL: Allow from VPC-B CIDR to target port. NACL allow both directions.
3. Verify TGW association: VPC-B attachment associated to correct TGW route table.
4. Check TGW routes: Is VPC-A CIDR present (static or propagated)? State = active.
5. Spoke route tables: Subnets in VPC-B have a route to VPC-A CIDR via TGW; VPC-A subnets route back to VPC-B via TGW.
6. DNS & MTU: Resolve to private IP; test with ping -s/ping -l (size) to detect fragmentation issues.
7. Re-test with bypass: Temporarily bypass inspection/firewall to isolate whether middlebox is the culprit.

# Requires EC2 Instance Connect or SSM Session Manager on the source host
# This uses SSM to run reachability checks without public SSH/RDP

$instanceId = "i-xxxxxxxxxxxx"
$dest = "10.1.20.15"
$port = 5432

Send-SSMCommand -InstanceId $instanceId -DocumentName "AWS-RunShellScript" -Parameter @{ commands = @(
  "date",
  "echo Testing to $dest:$port",
  "timeout 3 bash -c '

resource "aws_ec2_transit_gateway" "hub" {
  description = "Org TGW"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
  dns_support  = "enable"
  vpn_ecmp_support = "enable"
}

resource "aws_ec2_transit_gateway_route_table" "prod" {
  transit_gateway_id = aws_ec2_transit_gateway.hub.id
}

# Associate VPC attachments to prod table; add static routes if needed

{
  "Resources": {
    "S3GatewayEndpoint": {
      "Type": "AWS::EC2::VPCEndpoint",
      "Properties": {
        "ServiceName": { "Fn::Sub": "com.amazonaws.${AWS::Region}.s3" },
        "VpcId": { "Ref": "VPC" },
        "RouteTableIds": [{ "Ref": "PrivateRtA" }, { "Ref": "PrivateRtB" }],
        "VpcEndpointType": "Gateway"
      }
    }
  }
}

# Find drops from a specific source to a destination in VPC Flow Logs (Athena)
SELECT * FROM vpc_flow_logs
WHERE srcaddr = '10.0.12.34'
  AND dstaddr = '10.1.20.15'
  AND action = 'REJECT'
  AND start >= to_unixtime(current_timestamp - interval '1' hour);

# Identify subnets without explicit routes to TGW or IGW (potential isolation)
Get-EC2RouteTable | % {
  $rt = $_
  $hasTgwOrIgw = $rt.Routes | Where-Object { $_.GatewayId -like 'igw-*' -or $_.TransitGatewayId -like 'tgw-*' }
  if(-not $hasTgwOrIgw){ "Isolated? $($rt.RouteTableId)" }
}

# Find overlapping CIDRs between VPCs quickly
$vpcs = Get-EC2Vpc | Select-Object VpcId, CidrBlock
foreach($a in $vpcs){ foreach($b in $vpcs){
  if($a.VpcId -ne $b.VpcId){
    # naive check; for production use a proper CIDR overlap function
    if($a.CidrBlock.Split('/')[0] -eq $b.CidrBlock.Split('/')[0]){
      "Possible overlap pattern: $($a.VpcId) and $($b.VpcId) ($($a.CidrBlock) vs $($b.CidrBlock))"
    }
  }
}}

# Report VPN connections that are not 'available'
Get-EC2VpnConnection | Where-Object { $_.State -ne 'available' } | 
  Select-Object VpnConnectionId, State, TransitGatewayId, VpnGatewayId