Tailscale as a Zero-Trust Access Layer for Kubernetes

When you run services on a Kubernetes cluster in an isolated network, the question of access becomes critical. Who can reach what, and how do you expose services without punching holes in your firewall? This post covers how we use Tailscale as a zero-trust access layer alongside Cloudflare Tunnel for a dual public/private access model on K8S-CLUSTER.

The Two-Tier Access Model

Our cluster serves two audiences with different trust levels:

• Public (Cloudflare Tunnel) — Blog readers, URL shortener redirects, PDF tools. No authentication required for the service itself, but paths are restricted via Cloudflare Access policies.
• Private (Tailscale) — Admin dashboards, monitoring, password manager, management UIs. Only accessible to devices on the tailnet — zero public exposure.

Some services need both: A blog serves content publicly but its admin panel is Tailscale-only. A URL shortener handles public redirects via Cloudflare but the REST API is restricted to Tailscale. An analytics service accepts tracking events publicly but the dashboard is private.

Tailscale Kubernetes Operator

The Tailscale Kubernetes Operator (v1.94.2) runs in a dedicated tailscale namespace with privileged PodSecurity (the proxy pods need CAP_NET_ADMIN for WireGuard tunneling). When it detects an Ingress resource with ingressClassName: tailscale, it creates a StatefulSet with a proxy pod that joins the tailnet.

A ProxyClass resource centralizes configuration for all proxy pods:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: default
spec:
  tailscale:
    acceptRoutes: false
  statefulSet:
    pod:
      tailscaleContainer:
        resources:
          requests:
            memory: "32Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "200m"

Each proxy pod is lightweight — 32Mi memory request, 50m CPU. With several services exposed, the total Tailscale overhead is under 256Mi RAM and 400m CPU across the cluster.

The Ingress Pattern

Every Tailscale-exposed service follows an identical 17-line Ingress manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-tailscale
  namespace: my-app
  annotations:
    tailscale.com/proxy-class: "default"
spec:
  ingressClassName: tailscale
  defaultBackend:
    service:
      name: my-app
      port:
        name: http
  tls:
    - hosts:
        - k8s-my-app

The hostname in tls.hosts becomes the MagicDNS name: k8s-my-app.your-tailnet.ts.net. HTTPS certificates are automatically provisioned. No cert-manager, no Let's Encrypt configuration, no renewal cronjobs.

We currently expose services this way:

Network Policy Integration

The real power comes from combining Tailscale with CiliumNetworkPolicy. Each service explicitly declares which traffic sources are allowed. Here's a typical policy showing the dual-access pattern for a web application:

ingress:
  # Public access via Cloudflare Tunnel
  - fromEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: cloudflared
          app: cloudflared
    toPorts:
      - ports:
          - port: "8080"
            protocol: TCP
  # Private access via Tailscale
  - fromEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: tailscale
    toPorts:
      - ports:
          - port: "8080"
            protocol: TCP

The key insight: both cloudflared pods and Tailscale proxy pods are just regular pods in the cluster. CiliumNetworkPolicy treats them identically — match by namespace and label, allow on specific ports. The Tailscale ingress rule matches any pod in the tailscale namespace since the operator dynamically creates proxy pods with varying names.

For services that are Tailscale-only (like an admin dashboard), the policy simply omits the cloudflared rule:

ingress:
  # Tailscale only — no public access
  - fromEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: tailscale
    toPorts:
      - ports:
          - port: "8080"
            protocol: TCP

Cloudflare Tunnel: The Public Side

For comparison, the Cloudflare Tunnel deployment (cloudflared) takes the opposite approach. It runs with an egress-only network policy — no ingress at all. The tunnel pods initiate outbound connections to Cloudflare's edge servers and are explicitly allowed to reach only specific backend services:

egress:
  # Each backend service is explicitly listed
  - toEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: my-app
          app: my-app
    toPorts:
      - ports:
          - port: "8080"
            protocol: TCP
  # Repeat for each public service...

  # Cloudflare edge (tunnel connection)
  - toCIDR:
      - 198.41.192.0/24
      - 198.41.200.0/24
    toPorts:
      - ports:
          - port: "7844"

Adding a new public service means adding one egress rule to cloudflared's policy. Removing public access means removing the rule. The tunnel itself never changes — routing is configured in Cloudflare's dashboard.

The Critical Gotcha: Cilium eBPF + Tailscale

This is the single most important lesson from our implementation, and it's poorly documented:

Tailscale's Service annotation (tailscale.com/expose) does not work with Cilium's eBPF kube-proxy replacement.

When Cilium replaces kube-proxy, it intercepts ClusterIP DNAT at the traffic control (tc) eBPF layer. Tailscale's Service annotation model relies on symmetric routing through the Kubernetes Service abstraction. Cilium's eBPF interception creates asymmetric routing — the request arrives through the Tailscale proxy but the response bypasses it.

The fix: use Tailscale Ingress resources (L7) instead of Service annotations (L4). The Ingress model uses tailscale serve to reverse-proxy at Layer 7, which doesn't depend on Kubernetes Service DNAT. It just works.

We also needed bpf.hostLegacyRouting=true in Cilium because forwardKubeDNSToHost (which Talos enables by default) breaks eBPF host-routing.

ACL Configuration

The Tailscale ACL setup requires specific tag ownership:

• tag:k8s-operator must own tag:k8s — the operator assigns this tag to proxy pods
• autogroup:member gets accept rules for tag:k8s:* — tailnet users can reach services
• The OAuth client needs Devices (write) and Auth Keys (write) scopes

Watch out for naming: the Helm chart names the OAuth secret tailscale-operator but the Deployment is just operator. The inconsistency has tripped up many operators.

Other Gotchas

• HTTPS cert delay — MagicDNS hostnames resolve immediately but HTTPS certificates take 1-2 minutes to provision. Don't assume TLS is ready in the first 30 seconds.
• Container port, not Service port — CiliumNetworkPolicy rules must reference the pod's container port (e.g., 8080), not the Service port (e.g., 80). Cilium operates at the pod level.
• Privileged namespace required — The tailscale namespace needs privileged PodSecurity. Proxy pods require CAP_NET_ADMIN for WireGuard tunnel setup.

The Result

With this setup, adding a new service to the tailnet is three steps:

1. Create a 17-line Ingress manifest
2. Add a Tailscale ingress rule to the service's CiliumNetworkPolicy
3. kubectl apply

Within 2 minutes, the service is available at https://k8s-cluster-<name>.your-tailnet.ts.net with a valid HTTPS certificate, accessible only to tailnet members, and protected by both Tailscale ACLs and Kubernetes network policies.

No port forwarding. No firewall rules. No certificate management. No VPN client configuration. Just add an Ingress and it works.