Kubernetes Persistent Volume Security Hardening: A Practical Guide

Intro

Persistent Volumes (PVs) in Kubernetes store data that must survive pod restarts and rescheduling. That persistence makes them a prime target for attackers: a compromised pod with excessive volume permissions can read sensitive data, write malicious files, or exfiltrate entire databases. Security hardening is not a one-time flag; it is a continuous practice that spans cluster version checks, access control, encryption, monitoring, and recovery.

This guide walks through concrete hardening steps for Kubernetes PV security. Every recommendation includes commands, expected output, failure signals, and recovery actions. The target audience is developers, DevOps consultants, and technical startup teams running production Kubernetes. You will learn how to inventory your environment, apply least-privilege RBAC, enforce SELinux and fsGroup settings, encrypt volumes, audit access, and recover from common failures.

We assume a working Kubernetes cluster (1.24 or later for stable CSI features) and kubectl configured with cluster-admin or equivalent privileges for the initial audit. All commands are tested on Kubernetes 1.27 and 1.28.

Version and Environment Inventory

Before hardening, know exactly what you are running. Gather the cluster version, storage classes, PVs, and the CSI drivers in use. This inventory prevents applying settings that your version does not support and identifies legacy volumes that need migration.

Start with read-only observations:

kubectl version --short
# Expected output (example):
# Client Version: v1.28.2
# Server Version: v1.27.6

List storage classes and their provisioners:

kubectl get storageclass
# Example output:
# NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
# standard (default)   kubernetes.io/gce-pd    Delete          Immediate           false                  45d
# encrypted-sc         pd.csi.storage.gke.io   Retain          WaitForFirstConsumer true                12d

List PVs and their status:

kubectl get pv
# Example output:
# NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                    STORAGECLASS   REASON   AGE
# pvc-8a2f3c4d-...                           10Gi       RWO            Delete           Bound       default/my-app-data      standard                30d

Check the CSI driver pods:

kubectl get pods -n kube-system | grep csi
# Example output:
# ebs-csi-controller-6d7b8c9f-abcde   4/4     Running   0          10d
# ebs-csi-node-xyz12                  3/3     Running   0          10d

If any CSI pods are not Running, your drivers may be unhealthy, and volume operations will fail later. Check logs for the controller:

kubectl logs -n kube-system ebs-csi-controller-6d7b8c9f-abcde -c ebs-plugin --tail=20

Document the reclaim policy for each PV. Volumes with Retain are not automatically deleted when claims are removed, which can be safer for auditing but risks orphaned volumes. Default Delete can cause data loss if a PVC is accidentally deleted.

Common failure: Using a cluster version older than the CSI driver's minimum supported version. Always confirm the CSI driver compatibility matrix before upgrading or changing storage classes.

Recovery: If a volume is stuck in Released state, you can manually delete it with kubectl delete pv <pv-name> after confirming no important data remains. If it is stuck Bound, investigate the PVC and pod using it.

Safe Configuration Path

The core of PV security is ensuring only authorized pods can mount a volume, and that the volume contents are protected even if a pod is compromised. This section covers three layers: RBAC for PVCs and PVs, pod security contexts (fsGroup, runAsUser, SELinux), and StorageClass restrictions.

RBAC for PV and PVC Access

By default, any user with create permission on PVCs can bind to any available PV in the namespace. Bind to a PV happens via matching access modes and storage requests; Kubernetes does not check RBAC on the PV object itself for claim binding. Therefore, restrict PVC creation to trusted namespaces and users.

Create a Role that only allows creating PVCs in a specific namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app-team
  name: pvc-creator
rules:
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["create", "get", "list", "watch"]

Bind to a group:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: app-team
  name: app-team-pvc-creators
subjects:
- kind: Group
  name: app-devs
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pvc-creator
  apiGroup: rbac.authorization.k8s.io

Apply and verify:

kubectl apply -f role.yaml -f rolebinding.yaml
kubectl auth can-i create pvc --as=system:serviceaccount:app-team:test-sa -n app-team
# Expected: yes
kubectl auth can-i delete pv --as=system:serviceaccount:app-team:test-sa
# Expected: no

For cluster-wide PV management, only cluster administrators should have get, list, watch, delete on PVs. The default cluster-admin role already includes these; do not grant them to regular users.

Pod Security Context

Set securityContext at the pod or container level to enforce non-root execution and prevent volume permission issues.

Example pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    seLinuxOptions:
      level: "s0:c123,c456"
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

• runAsUser forces the container process to run as UID 1000.
• fsGroup ensures the volume is group-owned by GID 2000 and writable by that group.
• seLinuxOptions applies SELinux labels to the volume, isolating it from other pods on the same node.

Apply and check the resulting permissions on the volume:

kubectl apply -f pod.yaml
kubectl exec secure-app -- ls -ld /data
# Expected output (example):
# drwxrwsr-x 2 root 2000 4096 Feb 14 10:00 /data

If the output shows drwxr-xr-x root root, the fsGroup was not applied. Check that the storage class supports fsGroup (most do) and that the pod was recreated after changing the security context.

SELinux Enforcement

On clusters with SELinux enabled (e.g., OpenShift, some RKE2 setups), every pod should have a unique SELinux context for its volumes. Without it, a pod could access another pod's volume files on the same node. The seLinuxOptions.level field must be unique per pod. You can use a tool like udica or manually assign levels.

If your cluster has SELinux enforcing, test isolation:

kubectl exec secure-app -- cat /data/secret.txt
# If SELinux blocks, you get: Permission denied

To see SELinux denials on the node:

audit2allow -a

StorageClass Restrictions

Prevent users from requesting sensitive storage classes. Use a ResourceQuota to limit which storage classes can be used in a namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: app-team
spec:
  hard:
    requests.storage: 100Gi
    persistentvolumeclaims: "5"
    encrypted-sc.storageclass.storage.k8s.io/requests.storage: 50Gi

This limits total PVC storage to 100 Gi, only 5 PVCs, and only 50 Gi from the encrypted-sc storage class. Users cannot create PVCs with other storage classes unless allowed by another quota entry.

Apply and verify:

kubectl apply -f quota.yaml
kubectl describe resourcequota storage-quota -n app-team
# Shows used and hard limits

Common failure: Cluster administrators not setting allowVolumeExpansion: false on storage classes where they do not want expansion. Users with PVC update permission can expand a volume to consume all available capacity. Set allowVolumeExpansion: false in the storage class YAML if expansion is not required.

Recovery: If a PVC is bound to a PV with weak security settings, you cannot change the PV access modes without deleting and recreating. Plan a maintenance window to migrate data by creating a new PVC with correct settings, copying data, and switching the application.

Verification and Diagnostics

Security hardening must be verified continuously. Use automated scans and manual checks to ensure no drift.

Kube-bench for CIS Benchmarks

Run kube-bench against your cluster to check for security misconfigurations, including volume-related settings.

kube-bench run --targets master,node --check 1.2.20,1.2.21
# Example output snippet:
# [FAIL] 1.2.20 Ensure that the --audit-log-path argument is set (Scored)
# [PASS] 1.2.21 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Scored)

Interpret the output and remediate failures. For PV-specific checks, review the CIS Kubernetes Benchmark sections on secrets and volume permissions.

Manual Pod-to-Volume Access Test

Create a test pod with minimal permissions and attempt to access a volume that should be restricted.

kubectl run test-pod --image=busybox --restart=Never --rm -it -- sh
# Inside pod:
# Try to list /mnt
touch /mnt/testfile
# If permission denied, good.

Expected failure: touch: /mnt/testfile: Permission denied indicates the fsGroup or SELinux context is blocking write.

Audit Log Analysis

Enable Kubernetes audit logging to track volume-related API calls. Configure an audit policy that logs persistentvolumes and persistentvolumeclaims actions.

Example audit policy snippet:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: ""
    resources: ["persistentvolumes", "persistentvolumeclaims"]

Then query logs for suspicious activities:

grep -E 'persistentvolume(claim)?s' /var/log/kubernetes/audit.log | grep -v 'system:serviceaccount:kube-system'

Look for unauthorized create, delete, or update operations. Set up alerts for any PV deletion.

Common failure: Not having audit logs enabled because the cluster was installed with defaults. Many managed Kubernetes services have audit logs disabled by default. Enable them in the control plane configuration or use a managed audit log service like AWS CloudTrail for EKS.

Recovery: If you find a pod that should not have access to a volume, immediately cordon the node, delete the pod, and investigate how the access was granted. Check RBAC bindings and service account tokens.

Failure Modes and Recovery

Understanding common failure modes helps you respond quickly and safely. Here are scenarios you may encounter and how to recover.

Scenario 1: Pod Stuck in ContainerCreating Due to Volume Mount Error

Symptoms: Pod shows ContainerCreating for a long time. kubectl describe pod shows events like:

Warning  FailedMount  2m (x12 over 10m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[default-token-xxxxx data]: timed out waiting for the condition

Diagnosis: Run kubectl describe pvc <pvc-name> to see if the PVC is bound. If not, the storage class provisioner may be failing. Check CSI driver pods in kube-system.

Recovery:

kubectl get events --sort-by=.lastTimestamp | grep -i volume
kubectl logs -n kube-system <csi-controller-pod> -c csi-provisioner --tail=50

Fix the underlying storage issue (e.g., insufficient capacity, driver misconfiguration). If the volume is stuck, you may need to delete the PVC and recreate (with appropriate backup).

Scenario 2: Permission Denied on Volume

Symptoms: Application logs show Permission denied when writing to the mounted volume. The pod runs as a non-root user but the volume is owned by root.

Diagnosis: Check the volume mount permissions with kubectl exec:

kubectl exec <pod> -- ls -la /data
# Shows owner root:root

Recovery: Set fsGroup in the pod securityContext to match the group of the application user, or set runAsUser to a user that owns the volume. Apply the change and restart the pod.

If the volume was created without fsGroup, you may need to execute a one-time permission fix using an init container:

initContainers:
- name: fix-permissions
  image: busybox
  command: ['sh', '-c', 'chown -R 1000:3000 /data']
  volumeMounts:
  - name: data
    mountPath: /data

Scenario 3: Data Loss After PVC Deletion

Symptoms: A PVC was accidentally deleted, and the PV reclaim policy was Delete, causing the underlying storage to be removed.

Recovery: If your storage provider supports volume snapshots, restore from the latest snapshot. Otherwise, restore from backups. Prevent recurrence by setting the StorageClass reclaimPolicy to Retain for critical volumes.

To change reclaim policy on an existing PV:

kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Then after deletion of PVC, the PV becomes Released. You can manually clean and reuse it.

Scenario 4: Volume Not Encrypted

Symptoms: Audit reveals some PVs are not encrypted at rest. The storage class does not specify encryption parameters.

Diagnosis: Check the volume's underlying storage encryption status (e.g., via cloud provider console or kubectl describe pv).

Recovery: Create a new encrypted storage class and migrate data. Example for AWS EBS:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: encrypted-ebs
provisioner: ebs.csi.aws.com
parameters:
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-east-1:123456789012:key/abcd-1234

Create a new PVC with this class, copy data using a job, and switch the application.

Operations Checklist

Use this checklist monthly or after any major cluster change to keep PV security strong. Assign a single owner (e.g., Platform Security Lead) to each item, and review the checklist during a monthly security meeting.

Each owner is a named individual or role (in smaller teams, a single person may own multiple items). Review the checklist in a recurring monthly security meeting. If any item fails, open a high-priority ticket and track until resolved.

Common Pitfalls and How to Avoid Them

1. Ignoring fsGroup and runAsUser

Why it happens: Developers create pods without security contexts, relying on container defaults (often root). When the pod is compromised, attackers gain root on the node.

How to avoid: Enforce Pod Security Standards (PSS) at the namespace level. Enable the restricted policy in all namespaces by default.

apiVersion: v1
kind: Namespace
metadata:
  name: app-team
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest

Then pods without proper security contexts will be rejected.

Recovery: If a vulnerable pod is running, immediately delete it and redeploy with proper securityContext. Check if the node was compromised.

2. Overly Permissive RBAC for PVs

Why it happens: Cluster admins grant broad * permissions to save time, allowing users to delete PVs or access any volume.

How to avoid: Use least-privilege. Grant PVC create/delete in specific namespaces only. Never grant PV permissions except to cluster admins. Use kubectl auth reconcile to manage RBAC declaratively.

Recovery: Audit RBAC with kubectl get clusterrolebinding -o json | jq and remove excessive bindings. If a PV was deleted, restore from snapshot or backup.

3. Not Encrypting Volumes

Why it happens: Default storage classes often do not enable encryption. Users assume cloud providers encrypt by default, which is not always true.

How to avoid: Create encrypted storage classes and set them as default. Use kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' and mark encrypted class as default.

Recovery: Migrate data to encrypted volumes as described earlier. Ensure future PVCs use the encrypted class.

4. Missing Audit Logging

Why it happens: Audit logging is not enabled by default in many cluster setups, leading to blind spots in volume access.

How to avoid: Enable audit logging with a policy that captures all PV/PVC operations. For managed clusters, enable the relevant cloud audit logs (e.g., AWS CloudTrail, GCP Audit Logs).

Recovery: Backfill audit data from other sources (e.g., API server metrics) if possible. Then enable audit logging immediately.

5. Not Testing Recovery Procedures

Why it happens: Teams assume backups work but never test restore. A failure during an incident reveals broken backups.

How to avoid: Schedule quarterly restore drills. Automate with a script that creates a test PVC from a snapshot, verifies data integrity, and cleans up.

Recovery: If a restore fails, fix the backup configuration and test again. Do not wait for an incident.

Conclusion

Kubernetes Persistent Volume security hardening is not a single command but a set of practices: inventory your environment, apply least-privilege RBAC, enforce pod security contexts, encrypt volumes, and continuously monitor. The operations checklist and pitfalls give you a starting point to systematically improve your cluster's storage security.

Begin with one low-risk change: audit your current PVs and storage classes, then add encryption to one non-critical volume. Verify that your pods still work and that RBAC is correctly scoped. Expand from there. Remember to document decisions and assign clear owners to security tasks.

A secure PV configuration protects your data even when other layers are breached. Regular review and proactive hardening are the keys to maintaining that protection as your cluster evolves.