↓ Skip to main content

One Dash Between Isolated and Exposed: The AND/OR Trap in Kubernetes NetworkPolicy

1629 words·8 mins· loading · loading ·
Table of Contents

While I was writing the NetworkPolicies for my homelab’s databases, and learning how they work at the same time, I found something that unsettled me. Whether a database is reachable by one exporter pod or by every pod in the cluster can come down to one dash in the YAML. Both versions are valid. The API server accepts both without a warning. And they look almost the same in a pull request.

My first reaction was that this is a design flaw in Kubernetes. After digging in, I think it’s something more ordinary: a consequence of how YAML writes lists and maps. That’s also why KYAML, the stricter YAML style Kubernetes added in 1.34, is worth knowing about.


Two rules in the same file
#

Here are two real rules from databases.yaml in Project Aether. The namespace starts from default-deny; these open exactly what’s needed.

The first lets every pod in the databases namespace look up names, but only through cluster DNS:

YAML
egress:
  - to:
      - namespaceSelector:
          matchLabels: { kubernetes.io/metadata.name: kube-system }
        podSelector:
          matchLabels: { k8s-app: kube-dns }
    ports:
      - { protocol: UDP, port: 53 }
      - { protocol: TCP, port: 53 }

The second decides who may connect to PostgreSQL:

YAML
ingress:
  - from:
      - namespaceSelector:
          matchLabels: { kubernetes.io/metadata.name: apps }
      - podSelector:
          matchLabels: { app.kubernetes.io/name: db-bootstrap }
      - podSelector:
          matchLabels: { app.kubernetes.io/name: db-dump }
      - podSelector:
          matchLabels: { app.kubernetes.io/name: db-exporters }
    ports:
      - { protocol: TCP, port: 5432 }

They look like the same kind of rule. They aren’t:

  • The DNS rule has one entry under to, holding two selectors. That’s AND: pods labelled kube-dns that are in kube-system.
  • The PostgreSQL rule has four entries under from. That’s OR: any pod in the apps namespace, or the bootstrap, dump or exporter pods in the database’s own namespace.

Both are what I meant. The only thing telling them apart is whether podSelector starts with a -.


The rule, from the API
#

The NetworkPolicy documentation states it plainly. from and to are lists of peers. Within one peer, every selector must match (AND). Across peers, any one may match (OR). And the docs add a warning that says a lot: “Be careful to use correct YAML syntax.”

Two more facts make the difference dangerous:

  • A podSelector on its own selects pods in the policy’s own namespace.
  • A namespaceSelector on its own allows every pod in the namespaces it selects. And an empty selector, {}, matches everything.

kubectl describe shows how the API understood the two rules above, straight from my cluster:

Allowing egress traffic:
  To Port: 53/UDP
  To Port: 53/TCP
  To:
    NamespaceSelector: kubernetes.io/metadata.name=kube-system
    PodSelector: k8s-app=kube-dns

Allowing ingress traffic:
  To Port: 5432/TCP
  From:
    NamespaceSelector: kubernetes.io/metadata.name=apps
  From:
    PodSelector: app.kubernetes.io/name=db-bootstrap
  From:
    PodSelector: app.kubernetes.io/name=db-dump
  From:
    PodSelector: app.kubernetes.io/name=db-exporters

One To: block with both selectors inside is AND. Each separate From: block is a separate way in.


One character, two meanings
#

Suppose I want PostgreSQL’s metrics exporter to run in the monitoring namespace instead, and allow only that exporter in. The intended rule is AND:

YAML
from:
  - namespaceSelector:
      matchLabels: { kubernetes.io/metadata.name: monitoring }
    podSelector:
      matchLabels: { app.kubernetes.io/name: postgres-exporter }

Now add one dash, the kind of thing an editor, a merge or a tired evening produces:

Diff
 from:
   - namespaceSelector:
       matchLabels: { kubernetes.io/metadata.name: monitoring }
-    podSelector:
+  - podSelector:
       matchLabels: { app.kubernetes.io/name: postgres-exporter }

I ran both through the API server with kubectl apply --dry-run=server, which validates and parses a manifest without storing anything. Both were accepted, with no warning. The second now means: every pod in monitoring, or any pod labelled postgres-exporter in databases. Every log shipper, every metrics agent, and anything anyone ever deploys into monitoring can now open a connection to the database.

Two panels comparing the same NetworkPolicy written two ways. Top, one peer holding both a namespaceSelector for the monitoring namespace and a podSelector for postgres-exporter: only the exporter pod reaches PostgreSQL, while the log shipper and any pod deployed later are refused. Bottom, the same selectors split into two peers by one extra dash: every pod in the monitoring namespace reaches PostgreSQL. In KYAML the first is one object and the second is two.
The same two selectors, one dash apart. One peer is AND: only the exporter gets in. Two peers are OR: the whole namespace does.

The worst version of the same slip uses an empty namespace selector. The intent is “pods labelled db-client, in any namespace”:

YAML
from:
  - namespaceSelector: {}
    podSelector:
      matchLabels: { db-client: "true" }

Add the dash, and the first entry becomes namespaceSelector: {} on its own: every pod in every namespace. The database is now open to the whole cluster. Any compromised pod, an app with a remote-code bug, a debug container, a CI job, gets a direct TCP path to PostgreSQL, and the database password becomes the only thing in the way.

Wrong in the other direction fails loudly
#

The mistake can go both ways, and the two directions behave differently.

If I’d removed a dash from my real PostgreSQL rule, merging the bootstrap entry into the apps one, the rule would mean “bootstrap pods in the apps namespace”. The real bootstrap Job runs in databases, so it would lose access, and so would the nightly dump. That breaks something within a day, and someone notices.

Writing OR where you meant AND breaks nothing. Everything that should work still works. The only change is that more things can connect, and nothing tells you. A NetworkPolicy that’s too strict fails closed and loudly. One that’s too loose fails open and silently. That’s why this deserves more care than an ordinary typo.


A design flaw, or YAML?
#

At first it felt like Kubernetes had made a dangerous API. Thinking about it more, the API itself is consistent: a list of peers, and each peer is a set of conditions that must all hold. Written as JSON, nobody would confuse the two:

JSON
"from": [ { "namespaceSelector": {...}, "podSelector": {...} } ]
"from": [ { "namespaceSelector": {...} }, { "podSelector": {...} } ]

The trouble is YAML’s block style. A list item and a map key look almost identical, and the boundary between “one object with two keys” and “two objects” is a dash and two spaces of indentation. YAML has several traps of this kind, where a small change in whitespace or quoting gives a different but still valid document. NetworkPolicy is simply where this one hurts most, because the different document is a security rule.


KYAML makes the boundary visible
#

Kubernetes 1.34 added KYAML as a kubectl output format: alpha in 1.34 (behind KUBECTL_KYAML=true), and beta, on by default, from 1.35. You ask for it with -o kyaml. It’s a strict subset of YAML, so every KYAML document is still valid YAML, but it always writes maps in {} and lists in [], double-quotes strings, and allows comments and trailing commas.

According to KEP-5295 and the Kubernetes blog post introducing it, the motivation is YAML’s general pitfalls: indentation that silently changes meaning, and values coerced into other types (the “Norway problem”, where an unquoted NO becomes false). NetworkPolicy isn’t mentioned, so I wouldn’t claim KYAML was made for this case. But this is exactly the class of problem it targets: structure decided by indentation.

Here is the server’s own reading of the two exporter rules, printed as KYAML. First the AND version:

from: [{
  namespaceSelector: {
    matchLabels: {
      kubernetes.io/metadata.name: "monitoring",
    },
  },
  podSelector: {
    matchLabels: {
      app.kubernetes.io/name: "postgres-exporter",
    },
  },
}],

And the version with the extra dash:

from: [{
  namespaceSelector: {
    matchLabels: {
      kubernetes.io/metadata.name: "monitoring",
    },
  },
}, {
  podSelector: {
    matchLabels: {
      app.kubernetes.io/name: "postgres-exporter",
    },
  },
}],

That }, { is the whole story. In block YAML the difference is one character you have to look for. In KYAML it’s a visible break between two objects.


How I guard against it now
#

  1. Read the policy back as the API sees it. After applying, run kubectl describe networkpolicy, as the docs suggest. One From: block per way in; count them.

  2. Review in KYAML. kubectl get networkpolicy -n databases -o kyaml, or the same on a server dry-run of the change, shows exactly where each peer starts and ends.

  3. Write each peer on its own, in flow style. Plain YAML lets you write a peer as one inline map, which makes AND hard to break by accident:

    YAML
    from:
      - { namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } },
          podSelector:       { matchLabels: { app.kubernetes.io/name: postgres-exporter } } }
  4. Test the policy in CI. A Conftest (OPA) rule can refuse the most dangerous shape outright, a peer that is nothing but an empty namespace selector:

    REGO
    package networkpolicy
    
    deny contains msg if {
      input.kind == "NetworkPolicy"
      some rule in input.spec.ingress
      some peer in rule.from
      peer.namespaceSelector == {}
      not peer.podSelector
      msg := sprintf("%s: an ingress peer with only namespaceSelector: {} admits every pod in the cluster", [input.metadata.name])
    }
  5. Test from the pod’s point of view. When I added default-deny policies to Aether’s gateway and cert-manager namespaces, I ran a probe pod with each component’s labels and checked it could reach what the policy allows and nothing else (Pod Security, Honestly). The same probe, run from an unrelated namespace against the database port, is the test that catches an accidental OR: it should be refused.

  6. Start from default-deny. With a default-deny policy in the namespace, a rule that’s too loose still opens only the one port it names, instead of everything.


What I learned
#

  • In a NetworkPolicy, one peer is AND and two peers are OR, and in block YAML the difference is a single dash.
  • Too loose fails silently; too strict fails loudly. Put your review effort where the silent failure is.
  • Read the policy the way the API reads it, with kubectl describe or -o kyaml, not the way you meant to write it.
  • It isn’t really a Kubernetes design flaw. The API is consistent; YAML’s block syntax hides the structure. KYAML puts it back on the page.

Further reading
#