↓ Skip to main content

Gateway API on a Homelab: Envoy Gateway, MetalLB and an Internal CA on K3s

2250 words·11 mins· loading · loading ·
Table of Contents
Project Aether – Building a Homelab - This article is part of a series.
Part 10: This Article

Every Kubernetes cluster that serves HTTP needs something at the edge: a component that accepts connections, terminates TLS and routes requests to the right Service. For years the answer was the Ingress API, usually with ingress-nginx behind it. That era is ending. The Ingress API is frozen, with new features going into Gateway API instead, and the community ingress-nginx project was retired in March 2026.

So when I built Project Aether’s K3s cluster, I skipped Ingress entirely. K3s ships with Traefik and its own load balancer, and I disabled both at install time. The edge is Gateway API, implemented by Envoy Gateway, with MetalLB giving it an address on the home network and cert-manager running an internal certificate authority.

The gateway was built, programmed and serving TLS for weeks before it had anything to route. That turned out to be useful: six things broke while building it, and two traps showed up before any traffic did. The first apps arrived on 17 November, and the last section covers what changed when they did.


Why not Ingress
#

Ingress tried to describe everything in one resource: the listener, the TLS certificate and the routing rules. The spec covered only the basics, so every controller added its own features through annotations (nginx.ingress.kubernetes.io/rewrite-target, traefik.ingress.kubernetes.io/...). An Ingress written for one controller rarely works on another, and the annotations aren’t validated, so a typo just does nothing.

Gateway API splits the same job into typed resources, each owned by a different role:

Resource Who owns it What it says
GatewayClass Infrastructure provider Which controller implements gateways of this class
Gateway Cluster operator Which ports listen, with which certificates, and which namespaces may attach routes
HTTPRoute Application team Which hostnames and paths go to which Services

An application team can add or change routes without permission to touch ports or certificates. Features that used to be annotations, such as header matching, traffic splitting and redirects, are fields in the spec, validated by the API server.

K3s installs Traefik and ServiceLB (its built-in LoadBalancer implementation) by default. I didn’t want either running alongside the real edge, so the control-plane install disables both from the start:

Bash
sh -s - server \
  --cluster-init \
  --secrets-encryption \
  --disable traefik \
  --disable servicelb

MetalLB: giving the gateway an address
#

In a cloud, a Service of type LoadBalancer gets an address from the cloud provider. On bare metal with ServiceLB disabled, nothing allocates one, and the Service that Envoy Gateway creates would sit at EXTERNAL-IP <pending> forever.

MetalLB fills that gap. In L2 mode, it takes a pool of addresses on the home LAN and answers ARP requests for them: when a device asks “who has this address?”, a MetalLB speaker on one of the nodes replies “me”. If that node fails, another speaker takes over the claim.

The pool needed one piece of planning outside Kubernetes. MetalLB’s addresses must never be handed out by the router’s DHCP server too, or another device on the network could be given the gateway’s address. I shrank the router’s DHCP range and gave MetalLB a small block of six addresses outside it, with the first reserved for the gateway.

One detail surprised me. Envoy Gateway creates its Service with externalTrafficPolicy: Local, and MetalLB only announces an address from a node that has a ready endpoint for it. So the gateway address follows the Envoy proxy pod: whichever node runs the proxy answers for the VIP. A side effect of Local is that Envoy sees real client source addresses, which matters later for access logs and rate limiting.


The Gateway API objects
#

Envoy Gateway is installed by ArgoCD from its Helm chart (gateway-helm, v1.9.2). The class and the gateway are plain manifests in Git:

YAML
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: aether-proxy
    namespace: gateway-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: aether-gateway
  namespace: gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: aether-wildcard-cert
      allowedRoutes:
        namespaces:
          from: All

Both listeners report Accepted, ResolvedRefs and Programmed as True. The gateway has an address from MetalLB and the HTTPS listener presents a certificate. For its first few weeks, zero HTTPRoutes were attached.


TLS: an internal CA with cert-manager
#

The HTTPS listener needs a certificate. For a first version, cert-manager runs a small internal CA:

ClusterIssuer selfsigned-root-issuer        (selfSigned)
   └── Certificate aether-root-ca           (isCA: true, in namespace cert-manager)
         └── ClusterIssuer aether-ca-issuer (ca: aether-root-ca-secret)
               └── Certificate aether-wildcard-cert   (wildcard, in gateway-system)
                     └── Gateway listener https

A self-signed issuer can usefully sign exactly one thing, the CA. The CA issuer then signs everything else. The root CA’s Certificate lives in the cert-manager namespace for a reason that’s easy to miss: a ClusterIssuer of type ca reads its Secret from cert-manager’s own namespace. Put the root anywhere else and the issuer reports secret not found.


Six things that broke on the way
#

# Symptom Cause Fix
1 Gateway API CRDs permanently OutOfSync, with a prune in either app able to delete CRDs the other needed A separate gateway-crds Application and the Envoy Gateway chart both managed the same CRDs One owner: the chart owns every Gateway API CRD
2 Sync failed with metadata.annotations: Too long Client-side apply stores the whole object in an annotation capped at 262,144 bytes. Envoy Gateway’s envoyproxies CRD is about 1.4 MB ServerSideApply=true on the app
3 Server-side apply never took effect Replace=true was set alongside it, and ArgoCD gives Replace precedence Drop Replace, which is destructive anyway
4 cert-manager issuers failed: no such kind cert-manager’s chart defaults to crds.enabled: false crds.enabled: true
5 The Envoy proxy requested 512 Mi of memory with no limit and could land on any node The chart values shape the controller. The proxy is generated from an EnvoyProxy resource, and there wasn’t one, so it got upstream defaults An EnvoyProxy resource, referenced from the GatewayClass, that pins the proxy and sets limits
6 A recovery runbook’s kubectl get pods -n envoy-gateway-system returned nothing Envoy Gateway names the proxy envoy-<namespace>-<gateway>-<hash>. Mine is envoy-gateway-system-aether-gateway-…, which looks like upstream’s default namespace. The real namespace is gateway-system Fix the runbook, and read resource names carefully

Number 1 is worth expanding, because it applies to any chart that bundles CRDs. Helm treats a chart’s crds/ folder as one unit. Either the chart owns every CRD, or you manage all of them yourself and tell Helm to skip them. Splitting ownership between two Applications gives you two controllers arguing over the same objects forever.

Number 5 shows how much the defaults mattered here. The EnvoyProxy resource now pins the proxy to one worker and sets limits:

YAML
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: aether-proxy
  namespace: gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        pod:
          nodeSelector:
            kubernetes.io/hostname: k3s-worker-01   # data plane apart from the controller-heavy worker-02
        container:
          resources:
            requests:
              cpu: 50m
              memory: 96Mi    # measured 73 Mi idle (10 Nov)
            limits:
              cpu: 250m
              memory: 256Mi   # re-size after Phase 04 traffic

The proxy idles at 73 Mi, against a default request of 512 Mi. On a 12 GB laptop, that difference is a whole small service. The comment on the limit is deliberate: an idle proxy says nothing about a busy one, so the numbers get measured again once real traffic arrives.


Two traps found before any traffic
#

The root CA would have re-keyed itself on Christmas Day
#

Neither Certificate set a lifetime, so both got cert-manager’s defaults: 90 days, and, since cert-manager 1.18, a private-key rotation policy of Always. That’s right for a leaf certificate. For a root CA it’s a time bomb. On its first renewal, 25 December 2026, the root would have been reissued with a new key, and every device that trusted the old root would reject everything signed by the new one.

Nothing trusted the root yet, so nothing broke. But the moment I installed it on a laptop or phone, the clock would have started. The root now has a 10-year lifetime and keeps its key on renewal; the wildcard leaf keeps the defaults, because leaves should rotate (ADR-021):

YAML
  duration: 87600h       # 10 years
  privateKey:
    rotationPolicy: Never

The same review found a second problem with the certificate’s names. The wildcard covers *.aether.local, and .local is reserved for multicast DNS (RFC 6762). macOS, iOS and many Linux systems send .local lookups to mDNS rather than to the router’s DNS, so the same name resolves differently on different devices. The fix was to use a subdomain of a domain I own, with publicly trusted certificates from Let’s Encrypt via the DNS-01 challenge through Cloudflare. DNS-01 needs no inbound access, and with a public CA there’s no root to install on every device. That’s now done, and *.aether.local is retired (next section).

The first route would have timed out
#

The apps namespace has a default-deny NetworkPolicy. The first HTTPRoute would have been Accepted and Programmed, and every request through it would have timed out, because nothing allowed the Envoy proxy to reach pods in apps. That policy now exists, ahead of the first route, and it’s narrow: only the Envoy proxy pods in gateway-system may connect, not the controller or anything else in that namespace.

YAML
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: gateway-system
          podSelector:
            matchLabels:
              app.kubernetes.io/name: envoy
              app.kubernetes.io/component: proxy

The namespaceSelector and podSelector sit in one from entry, so a connection must match both. As two separate entries they’d mean “either”, which would let every pod in gateway-system in, plus every Envoy-labelled pod in any namespace.


Then the first apps arrived
#

On 17 November the gateway got its first real traffic: a developer toolbox, a notes app and a knowledge base moved onto the cluster. Three things changed at the edge to make that work (ADR-021 addendum).

Real names and real certificates. cert-manager now has a Let’s Encrypt ClusterIssuer that solves the DNS-01 challenge through Cloudflare, with an API token that can only edit DNS in that one zone. The HTTPS listener serves a single ECDSA wildcard for *.lab.chetan-thapliyal.cloud. Any device trusts it without installing anything, so the internal CA is left for internal needs only, which is what the 10-year root is for.

Reachable over the tailnet, from anywhere. The Tailscale operator publishes the Envoy data plane as one tailnet node, so my devices reach the gateway the same way at home and away. One public DNS record, *.lab, points at that node’s tailnet address. Anyone can resolve it, but only tailnet members can connect to it, so the rule became “no public records that point at anything reachable from the internet”. No split DNS, no resolver to run, no inbound port.

Routes, owned by the apps. Each app carries its own HTTPRoute next to its Deployment, attached to the shared gateway’s HTTPS listener. This is the role split from the start of the post, working as intended: the app says which hostname goes to which Service, and never touches ports or certificates.

YAML
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: it-tools
spec:
  parentRefs:
    - name: aether-gateway
      namespace: gateway-system
      sectionName: https   # HTTP is redirected for every lab name (gateway/http-redirect.yaml)
  hostnames:
    - it-tools.lab.chetan-thapliyal.cloud
  rules:
    - backendRefs:
        - { name: it-tools, port: 80 }

Two smaller routes belong to the platform. One redirects plain HTTP to HTTPS for every lab name. The other, ping.lab…, is answered by Envoy itself with a fixed response and no backend pod. A single request to it tests DNS, the tailnet path, the certificate and the gateway together, so it’s what the outside monitor checks. The NetworkPolicy from the previous section was already in place when the first route attached.

How a request reaches an app. A phone on the mesh asks public Cloudflare DNS for the app's name, and a wildcard record answers with the gateway's mesh address. The phone opens HTTPS over the private mesh to the Tailscale proxy in the gateway-system namespace, which forwards TCP to Envoy Gateway. Envoy terminates HTTPS with the Let's Encrypt wildcard certificate and calls the app pod picked by the HTTPRoute; the pod reaches PostgreSQL or Redis only if its labels opt in. cert-manager renews the wildcard by adding a TXT record in Cloudflare DNS. Network policies deny by default.
One request, end to end: the name is public, the address only works on the mesh, and every hop inside the cluster is denied unless it opts in.

What I learned
#

  • Every CRD needs exactly one owner. Two Applications managing the same CRDs fight forever.
  • Large CRDs need server-side apply, and Replace=true silently cancels it.
  • Chart values shape the controller, not always the workload it generates. Check what the generated pods actually request.
  • A root CA is not a leaf. Defaults that are right for leaves (short lifetime, new key on every renewal) are dangerous for a root.
  • Don’t use .local for unicast DNS.
  • Pre-position the network policy before the first route, or the first deploy fails in a way that looks like an application bug.

Architecture takeaways
#

Why Gateway API replaced Ingress
#

Ingress put three roles’ concerns into one object, and left everything beyond the basics to vendor annotations. Gateway API makes the split explicit: the infrastructure provider owns the class, the operator owns listeners and certificates, and application teams own routes. Each role can change its part without permission over the others, and features are typed fields the API server validates, not strings it ignores.

Why LoadBalancer needs a controller off-cloud
#

A Service of type LoadBalancer is a request, not a mechanism. In a cloud, the cloud controller manager fulfils it by creating a real load balancer. On bare metal, something else has to own a pool of addresses and make the network deliver traffic for them, which is MetalLB’s job here. Kubernetes leaves that to a pluggable controller on purpose: how an address reaches a node is a property of the network, which the cluster doesn’t control.


Previously in the series: Seven Silent GitOps Gotchas, and what ArgoCD and Helm didn’t tell me. Every part is listed on the series page.

Project Aether – Building a Homelab - This article is part of a series.
Part 10: This Article

Related