Multi-tenant Kubernetes platforms enable multiple teams to share infrastructure while maintaining security isolation. This guide explores strategies for implementing strong multi-tenancy, from basic namespace isolation to advanced techniques like virtual clusters and sandboxed containers.

Multi-Tenancy Isolation Strategies: Securing Kubernetes Platforms for Multiple Teams

Multi-Tenancy Models

Namespace-Based Multi-Tenancy

┌─────────────────────────────────────────────────────────┐
│            Shared Kubernetes Cluster                     │
│                                                          │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐       │
│  │ Namespace  │  │ Namespace  │  │ Namespace  │       │
│  │  Team A    │  │  Team B    │  │  Team C    │       │
│  │  (Dev)     │  │  (Staging) │  │  (Prod)    │       │
│  │            │  │            │  │            │       │
│  │ [Pods]     │  │ [Pods]     │  │ [Pods]     │       │
│  │ [Services] │  │ [Services] │  │ [Services] │       │
│  └────────────┘  └────────────┘  └────────────┘       │
│                                                          │
│  Shared: Control Plane, Nodes, Network, Storage        │
└─────────────────────────────────────────────────────────┘

Advantages:

  • Simple to implement and manage
  • Efficient resource utilization
  • Built-in Kubernetes primitives

Limitations:

  • Limited isolation (shared kernel)
  • Potential noisy neighbor issues
  • Requires strong RBAC configuration

Virtual Cluster Multi-Tenancy

┌─────────────────────────────────────────────────────────┐
│            Host Kubernetes Cluster                       │
│                                                          │
│  ┌─────────────────────┐  ┌─────────────────────┐     │
│  │  Virtual Cluster A  │  │  Virtual Cluster B  │     │
│  │  ┌────────────────┐ │  │  ┌────────────────┐ │     │
│  │  │  API Server    │ │  │  │  API Server    │ │     │
│  │  │  Controller Mgr│ │  │  │  Controller Mgr│ │     │
│  │  │  Scheduler     │ │  │  │  Scheduler     │ │     │
│  │  └────────────────┘ │  │  └────────────────┘ │     │
│  │                     │  │                     │     │
│  │  [Namespaces]      │  │  [Namespaces]      │     │
│  │  [Pods]            │  │  [Pods]            │     │
│  └─────────────────────┘  └─────────────────────┘     │
│                                                          │
│  Shared: Nodes, Network, Storage                        │
└─────────────────────────────────────────────────────────┘

Advantages:

  • Strong isolation (separate control planes)
  • Full Kubernetes API per tenant
  • Custom CRDs per tenant

Limitations:

  • Higher overhead
  • More complex management
  • Cost of control plane per tenant

RBAC Configuration

Namespace-Scoped RBAC

# Namespace with labels for RBAC
apiVersion: v1
kind: Namespace
metadata:
  name: team-payments-prod
  labels:
    team: payments
    environment: production
    sensitivity: high

---
# Service Account for team
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-team-sa
  namespace: team-payments-prod

---
# Role with specific permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: team-developer
  namespace: team-payments-prod
rules:
# Allow managing application resources
- apiGroups: ["apps", ""]
  resources: ["deployments", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

# Allow viewing pods and logs
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]

# Allow exec for debugging
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

# Deny dangerous operations
- apiGroups: [""]
  resources: ["nodes"]
  verbs: []  # No access to nodes

---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-developer-binding
  namespace: team-payments-prod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: team-developer
subjects:
- kind: Group
  name: payments-developers
  apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
  name: payments-team-sa
  namespace: team-payments-prod

---
# Cluster-level read-only for discovery
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tenant-cluster-viewer
rules:
# Allow viewing namespaces
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]

# Allow viewing nodes (read-only)
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]

# Allow viewing custom resources definitions
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: payments-team-cluster-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tenant-cluster-viewer
subjects:
- kind: Group
  name: payments-developers
  apiGroup: rbac.authorization.k8s.io

Hierarchical Namespaces

# Hierarchical Namespace Controller (HNC) configuration
apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchyConfiguration
metadata:
  name: hierarchy
  namespace: team-payments
spec:
  parent: organization-engineering

---
# Subnamespace anchor
apiVersion: hnc.x-k8s.io/v1alpha2
kind: SubnamespaceAnchor
metadata:
  name: prod
  namespace: team-payments
spec:
  # Automatically creates team-payments-prod namespace

---
# Propagated RBAC
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: team-viewer
  namespace: team-payments
  annotations:
    hnc.x-k8s.io/propagate: "true"
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "watch"]

Network Isolation

Network Policies

# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: team-payments-prod
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

---
# Allow ingress from ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
  namespace: team-payments-prod
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: frontend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080

---
# Allow inter-namespace communication for specific services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-api-gateway
  namespace: team-payments-prod
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: payment-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          team: api-gateway
      podSelector:
        matchLabels:
          app.kubernetes.io/name: gateway
    ports:
    - protocol: TCP
      port: 8080

---
# Egress policy for external services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-external
  namespace: team-payments-prod
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: payment-service
  policyTypes:
  - Egress
  egress:
  # Allow DNS
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
  
  # Allow specific external APIs
  - to:
    - ipBlock:
        cidr: 52.1.2.3/32  # Stripe API
  - to:
    - ipBlock:
        cidr: 54.2.3.4/32  # PayPal API
    ports:
    - protocol: TCP
      port: 443

---
# Deny cross-tenant communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-other-tenants
  namespace: team-payments-prod
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          team: payments
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          team: payments
  - to:  # Allow platform services
    - namespaceSelector:
        matchLabels:
          platform-service: "true"

Resource Quotas and Limits

# Resource quota for namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-payments-prod
spec:
  hard:
    # Compute resources
    requests.cpu: "50"
    requests.memory: 100Gi
    limits.cpu: "100"
    limits.memory: 200Gi
    
    # Storage
    persistentvolumeclaims: "10"
    requests.storage: 500Gi
    
    # Objects
    pods: "100"
    services: "20"
    services.loadbalancers: "5"
    services.nodeports: "0"  # Disable NodePort
    
    # Secrets and ConfigMaps
    configmaps: "50"
    secrets: "50"

---
# Limit range for pod defaults
apiVersion: v1
kind: LimitRange
metadata:
  name: resource-limits
  namespace: team-payments-prod
spec:
  limits:
  # Container limits
  - type: Container
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    max:
      cpu: "4"
      memory: "8Gi"
    min:
      cpu: "50m"
      memory: "64Mi"
  
  # Pod limits
  - type: Pod
    max:
      cpu: "8"
      memory: "16Gi"
  
  # PVC limits
  - type: PersistentVolumeClaim
    min:
      storage: "1Gi"
    max:
      storage: "100Gi"

---
# Priority class for tenant
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: payments-production
value: 1000
globalDefault: false
description: "Priority for payments team production workloads"

Pod Security

# Pod Security Standard enforcement
apiVersion: v1
kind: Namespace
metadata:
  name: team-payments-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

---
# Security Context Constraints
apiVersion: v1
kind: SecurityContextConstraints
metadata:
  name: tenant-restricted
allowPrivilegedContainer: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
allowedCapabilities: []
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret
runAsUser:
  type: MustRunAsNonRoot
seLinuxContext:
  type: MustRunAs
fsGroup:
  type: MustRunAs
supplementalGroups:
  type: MustRunAs
seccompProfiles:
- runtime/default

---
# OPA Gatekeeper constraint
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPAllowedUsers
metadata:
  name: must-run-as-nonroot
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaceSelector:
      matchLabels:
        team: payments
  parameters:
    runAsUser:
      rule: MustRunAsNonRoot

Runtime Isolation

gVisor Sandboxing

# RuntimeClass for gVisor
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

---
# Pod using gVisor
apiVersion: v1
kind: Pod
metadata:
  name: sandboxed-app
  namespace: team-payments-prod
spec:
  runtimeClassName: gvisor
  containers:
  - name: app
    image: payment-processor:v1.0
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL

Kata Containers

# RuntimeClass for Kata Containers
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata

---
# Pod using Kata
apiVersion: v1
kind: Pod
metadata:
  name: isolated-workload
  namespace: team-payments-prod
spec:
  runtimeClassName: kata
  containers:
  - name: app
    image: sensitive-app:v1.0
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "2Gi"
        cpu: "1"

Virtual Clusters (vcluster)

# vcluster deployment
apiVersion: v1
kind: Namespace
metadata:
  name: vcluster-payments

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vc-payments
  namespace: vcluster-payments

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vc-payments
  namespace: vcluster-payments
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets", "services", "pods", "pods/attach", "pods/portforward", "pods/exec", "persistentvolumeclaims"]
  verbs: ["create", "delete", "patch", "update", "get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["statefulsets", "deployments", "replicasets"]
  verbs: ["create", "delete", "patch", "update", "get", "list", "watch"]

---
apiVersion: v1
kind: Service
metadata:
  name: payments-vcluster
  namespace: vcluster-payments
spec:
  type: ClusterIP
  ports:
  - name: https
    port: 443
    targetPort: 8443
  selector:
    app: vcluster
    release: payments

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: payments-vcluster
  namespace: vcluster-payments
spec:
  serviceName: payments-vcluster-headless
  replicas: 1
  selector:
    matchLabels:
      app: vcluster
  template:
    metadata:
      labels:
        app: vcluster
    spec:
      serviceAccountName: vc-payments
      containers:
      - name: vcluster
        image: rancher/k3s:v1.28.2-k3s1
        command:
        - /bin/k3s
        args:
        - server
        - --write-kubeconfig=/data/k3s-config/kube-config.yaml
        - --data-dir=/data
        - --disable=traefik,servicelb,metrics-server,local-storage
        - --disable-network-policy
        - --disable-agent
        - --disable-scheduler
        - --disable-cloud-controller
        - --flannel-backend=none
        - --kube-controller-manager-arg=controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 5Gi

Monitoring and Audit

# Audit policy
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log all requests at Metadata level
- level: Metadata
  omitStages:
  - RequestReceived

# Log pod changes at Request level
- level: Request
  verbs: ["create", "update", "patch", "delete"]
  resources:
  - group: ""
    resources: ["pods", "services"]

# Log RBAC changes at RequestResponse level
- level: RequestResponse
  verbs: ["create", "update", "patch", "delete"]
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

# Log secret access
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets"]
  verbs: ["get", "list", "watch"]

---
# FalcoI rules for runtime security
- rule: Unauthorized Process in Container
  desc: Detect execution of unexpected processes
  condition: >
    spawned_process and
    container and
    not proc.name in (allowed_processes)
  output: >
    Unauthorized process started in container
    (user=%user.name command=%proc.cmdline container=%container.name)
  priority: WARNING

- rule: Write Below Root
  desc: Detect writes to root filesystem
  condition: >
    write and
    container and
    fd.name startswith "/"
  output: >
    File opened for writing below root directory
    (user=%user.name command=%proc.cmdline file=%fd.name container=%container.name)
  priority: ERROR

Best Practices

Isolation Strategy

  1. Defense in Depth: Multiple layers of isolation
  2. Least Privilege: Minimal RBAC permissions
  3. Network Segmentation: Default-deny network policies
  4. Resource Limits: Enforce quotas and limits
  5. Runtime Security: Consider sandboxed runtimes for sensitive workloads

Operations

  1. Automated Provisioning: Self-service tenant creation
  2. Policy as Code: GitOps for RBAC and policies
  3. Monitoring: Track cross-tenant access attempts
  4. Audit Logging: Comprehensive audit trail
  5. Regular Reviews: Periodic security assessments

Scaling

  1. Start Simple: Begin with namespace-based multi-tenancy
  2. Measure Impact: Track noisy neighbor issues
  3. Gradual Enhancement: Add isolation as needed
  4. Consider Virtual Clusters: For strong isolation requirements
  5. Evaluate Trade-offs: Balance isolation vs. operational complexity

Conclusion

Multi-tenancy in Kubernetes requires careful planning and multiple isolation mechanisms. Success factors include:

  • Appropriate Model: Match isolation level to requirements
  • Strong Defaults: Secure-by-default configurations
  • Comprehensive Testing: Validate isolation boundaries
  • Continuous Monitoring: Detect isolation violations
  • Regular Updates: Keep security policies current

The goal is enabling safe resource sharing while maintaining security boundaries appropriate for your organization’s risk tolerance.