Kubernetes NetworkPolicy: roll out default deny without breaking production

Kubernetes NetworkPolicy default deny is a namespace-level safety baseline that blocks connections unless an allow rule explicitly preserves them. It matters because a compromised Pod should not automatically reach every workload or external destination. Platform, security and application teams should introduce it as a measured traffic-contract migration: prove that the CNI enforces NetworkPolicy, inventory real flows, add explicit DNS, ingress, telemetry and dependency paths, canary one namespace, and test both allowed and forbidden connections before expanding the policy.

Kubernetes documents NetworkPolicy as a Layer 3/4 control for traffic involving Pods. The NetworkPolicy concept guide also makes a crucial prerequisite explicit: creating the object has no effect unless the cluster's network plugin implements enforcement. This guide separates those documented semantics from rollout recommendations. Exact behavior around address translation, host-network traffic and existing connections can vary by implementation, so verify the CNI and managed-cluster documentation for the platform you operate.

Understand the security boundary before writing YAML

A NetworkPolicy selects Pods in its own namespace. Ingress and egress isolation are independent. A Pod is non-isolated in a direction until at least one policy selects it for that direction; after that, only traffic admitted by the union of matching allow rules can establish a connection. Policies are additive, so there is no first-match order. For Pod-to-Pod traffic, source egress and destination ingress must both permit the connection.

The control is not a firewall for every Kubernetes object. It does not authorize API requests, verify application identity, inspect HTTP paths or make TLS decisions. The upstream documentation lists other gaps in the standard API, including explicit deny priorities, network-event logging and policies that target Services by name. Use RBAC and workload identity for authorization, TLS or a service mesh for peer authentication where required, and the CNI's own observability for accepted and dropped flows.

An empty podSelector matches every Pod in the policy namespace. Empty ingress and egress rule lists then allow nothing in those directions. That makes default deny compact, but not self-sufficient: the baseline is useful only when the required allow policies exist and are tested.

Map the traffic contract before isolation

Start with a dependency matrix built from manifests, service ownership and observed flows. For each workload record the source identity, destination identity, protocol, destination port, direction, purpose, data sensitivity and owner. Include startup and background behavior, not only the happy-path HTTP request.

The minimum inventory normally includes DNS, ingress controllers or Gateway implementations, same-namespace services, cross-namespace services, databases, queues, identity providers, object storage, package or license endpoints, metrics scraping, trace and log collectors, admission or sidecar dependencies, and administrative probes. Record whether a destination has stable Pod and namespace labels. A Service ClusterIP is a virtual address; a policy usually describes the selected Pods or an IP block, and address translation timing can differ among networking implementations.

Do not turn an observed-flow export directly into policy. Observation can include attacker traffic, one-off debugging, old clients and accidental dependencies. Reconcile every flow with the owning team's declared contract. Conversely, a short observation window can miss month-end jobs, certificate renewal, failover paths and disaster-recovery calls.

Design stable identities with selectors

Prefer namespaceSelector and podSelector over mutable IP ranges for in-cluster dependencies. The control plane adds the immutable kubernetes.io/metadata.name label to namespaces, which can target a namespace by its current name. Organization-owned labels such as security.nooryasser.com/zone=ingress and app.kubernetes.io/name=checkout create clearer policy intent when their assignment is protected by admission and RBAC.

Selector indentation changes meaning. Two separate entries in a from or to list are alternatives. A single peer containing both namespaceSelector and podSelector requires both conditions. The NetworkPolicy API reference defines peer entries as OR, while the fields inside one peer combine to select matching Pods inside matching namespaces.

Use ipBlock for destinations that really are address ranges, such as a controlled private network or fixed external endpoint. Do not assume it will reliably identify traffic after LoadBalancer, Service or egress-gateway translation. Document whether the policy sees pre-NAT or post-NAT addresses on the actual CNI and cloud integration.

Roll out default deny as a migration

Choose a low-risk namespace with a known owner and representative traffic. Confirm enforcement first with a disposable test namespace: create two Pods, apply a narrow policy, and prove that a previously working connection fails while an allowed one succeeds. An accepted Kubernetes object is not evidence that packets are filtered.

Next, add allow policies without isolation and review their selectors. Because a non-isolated Pod still permits all traffic, these objects can be prepared safely, but they do not prove correctness. Apply default-deny ingress to the canary namespace, run the connection matrix and watch user-facing error rate, DNS failures, resets and CNI drop telemetry. Stabilize ingress before enabling default-deny egress; outbound dependencies are usually less visible and more dynamic.

Expand by workload class, not by blind namespace count. Keep a versioned exception register with owner, reason, scope and expiry. Rollback should remove or narrow the selecting isolation policy through a reviewed change; avoid a permanent allow-all policy that silently becomes the real baseline. Kubernetes notes that enforcement can converge asynchronously and that behavior for already-established connections is implementation-defined, so test new connections during rollout rather than relying on a warm connection pool.

Preserve DNS, ingress, probes and telemetry

DNS is the first common failure after egress isolation. Allow both UDP and TCP destination port 53 to the actual DNS Pods or addresses used by the cluster. The namespace and Pod labels differ by distribution, and NodeLocal DNSCache may put the resolver on a node-local address. Inspect Pod resolv.conf and the platform's DNS design instead of copying a kube-system selector blindly. The related Kubernetes DNS reliability guide explains how to trace the resolver path before changing it.

Ingress rules must allow the real source seen by the destination Pod. That may be an ingress-controller Pod, a Gateway data plane, a node address, a cloud load balancer range or a combination after source-address translation. Confirm it with CNI flow data. A rule that selects every Pod in an ingress namespace is wider than a peer that also requires the controller's protected Pod label.

HTTP, TCP and gRPC probes are initiated by the kubelet or implementation-specific node path rather than by an ordinary monitoring Pod. The NetworkPolicy specification always permits traffic from a Pod's node, but hostNetwork and node-address behavior has implementation caveats. Test readiness, liveness and startup probes after isolation on every node pool. See the probe reliability guide for separating traffic admission from process recovery.

Telemetry is bidirectional in architecture even when each connection is one-way. Prometheus may scrape the workload, while an OpenTelemetry collector may receive egress from it. Logs may leave through a sidecar or node agent. Write distinct rules for the actual initiator, and verify dashboards remain fresh after the policy is applied. A green application SLO with missing security telemetry is not a successful rollout.

Build an explicit baseline and narrow allows

The following example isolates every Pod in the payments namespace, restores cluster DNS using labels that must be checked on the target cluster, admits HTTPS only from the ingress controller, and permits the API Pods to reach PostgreSQL Pods in the data namespace. It is illustrative rather than portable configuration.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
  ingress: []
  egress: []
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-required-paths
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: payments-api
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-system
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-controller
      ports:
        - { protocol: TCP, port: 8443 }
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: data
          podSelector:
            matchLabels:
              app.kubernetes.io/name: postgres
      ports:
        - { protocol: TCP, port: 5432 }

Notice that ingress and egress are explicit, and that the namespace and Pod selectors for each peer share one list item. Validate namespaced labels through policy-as-code. If application teams can relabel arbitrary namespaces or privileged Pods, the selector becomes a self-service bypass.

Test positive and negative paths

A positive test proves each required edge: ingress to the exposed port, DNS over UDP and TCP, database access, telemetry delivery and declared external APIs. A negative test is equally important: attempt a wrong port, an unlabeled Pod, an unrelated namespace, a prohibited internet destination and an unexpected lateral connection. Run the matrix from clean Pods so existing sockets and DNS caches do not hide the result.

Observe at three levels. The application shows user impact and dependency errors. The CNI shows drops and the identity or rule involved. Synthetic probes prove the intended graph independently of production traffic. Alert on sustained policy drops tied to a service objective, but retain sampled evidence for investigation. The standard NetworkPolicy API does not provide network-event logs, so make observability a CNI-selection and operations requirement.

Test policy deletion, selector change, Pod restart, node replacement and CNI agent disruption. The Kubernetes lifecycle documentation warns that policy application is eventually handled by the network plugin; a newly created Pod can briefly see connectivity different from the intended final state. Workloads should retry bounded startup dependencies, and release checks should wait for real connectivity rather than only for the API object to exist.

Best practices and anti-patterns

Keep one policy purpose per object and use descriptive names. Protect namespace and workload labels. Review broad CIDRs and empty selectors. Require an owner and expiry for exceptions. Validate manifests server-side, test them on the same CNI version, and include the connectivity contract in the application release. Treat DNS, observability and break-glass access as designed dependencies, not emergency afterthoughts.

Avoid one giant policy generated from every observed flow. Avoid allow-all egress as a final state. Avoid selecting an entire system namespace when only one controller needs access. Avoid treating successful kubectl apply as enforcement proof. Avoid assuming NetworkPolicy authenticates the caller or understands an HTTP route. And do not use a policy change as the first time a team discovers its runtime dependencies.

When NetworkPolicy is not enough

NetworkPolicy is the right control for reducing Layer 3/4 reachability between Pods and network ranges. It is insufficient when the decision depends on user identity, HTTP method, API object, certificate identity, data classification or business authorization. Add application authorization, mTLS/service-mesh policy, an egress proxy, cloud firewall or database controls according to the boundary.

It may also be premature to enforce egress in a namespace whose dependencies are unknown and unobservable. First instrument and assign ownership; otherwise the policy becomes an outage generator followed by a permanent allow-all exception. In a development sandbox, a simpler ingress-only baseline may be proportionate, but production namespaces handling sensitive data should move toward explicit bidirectional contracts. The Kubernetes security checklist is a useful broader review alongside RBAC, Pod security, secrets and audit controls.

The operational rule is simple: default deny is the destination, not the first step. Prove enforcement, map and approve required edges, preserve platform traffic explicitly, canary each direction, test denial as carefully as success, and expand only with observable rollback. That turns NetworkPolicy from a YAML checkbox into a maintained production security contract.

Prepared by: Noor Yasser