Kubernetes Storage Class Security Hardening with Practical Examples

Intro

Storage classes in Kubernetes define how persistent volumes are provisioned. They also carry security-sensitive settings: provisioner credentials, filesystem options, mount parameters, and the access modes that determine who can claim a volume. A misconfigured storage class can expose data, allow privilege escalation, or lock teams out of their own storage. This article walks through security hardening for storage classes with practical examples, commands, and verification steps. It is written for developers, DevOps engineers, and technical startup teams who manage Kubernetes clusters and need to move from “we saw a problem” to “we verified the fix.”

The goal is operational safety. We observe before changing, limit the blast radius of each change, use placeholders instead of real secrets, verify every result, and document recovery paths. Each section includes commands you can run in a test namespace, expected output, failure signals, and a recovery decision.

Version and Environment Inventory

Before hardening anything, know your cluster version, the storage provisioner, and the CSI driver in use. Different Kubernetes versions and CSI drivers support different security features.

Run this read-only check first:

kubectl version --short
kubectl get storageclass
kubectl get csidriver

Expected output from kubectl get storageclass might look like:

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
gp2 (default)        kubernetes.io/aws-ebs   Delete          Immediate           true                   87d
fast-rbd             rbd.csi.ceph.com       Delete          Immediate           true                   12d

If your provisioner is a CSI driver, check its version and supported features with kubectl describe csidriver <driver-name>. For example, kubectl describe csidriver fast-rbd may show attachable: true, podInfoOnMount: true, and volumeLifecycleModes: [Persistent]. These flags affect how securely the driver can be configured.

The smallest change in this phase is often just documenting the environment. Create an inventory file in your ops repository:

# storage-inventory.md
Kubernetes version: 1.26.3
Provisioners: kubernetes.io/aws-ebs, rbd.csi.ceph.com
CSI drivers: ebs.csi.aws.com (v1.19.0), rbd.csi.ceph.com (v3.8.1)
Default storage class: gp2

Protect this inventory like any other internal document. It contains no secrets, but it tells an attacker exactly which storage drivers to target.

Safe Configuration Path

The safest configuration path is: create a new storage class rather than modifying an existing one, test it in a dedicated namespace, then slowly migrate workloads. Never change the default storage class in place without a rollback plan.

Example: suppose you want to harden the gp2 storage class to use encryption by default. Instead of editing the existing class, create a new one called gp2-encrypted:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-encrypted
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Save as gp2-encrypted.yaml and apply:

kubectl apply -f gp2-encrypted.yaml
kubectl get storageclass gp2-encrypted

Expected output:

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2-encrypted   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   5s

Now test it in a non-production namespace:

kubectl create namespace storage-test
kubectl apply -n storage-test -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-encrypted-pvc
spec:
  storageClassName: gp2-encrypted
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Check the PVC status:

kubectl get pvc -n storage-test test-encrypted-pvc

Expected output:

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
test-encrypted-pvc  Bound    pvc-1a2b3c4d-...                          1Gi        RWO            gp2-encrypted   10s

If the PVC remains Pending, describe it to see why:

kubectl describe pvc -n storage-test test-encrypted-pvc

Common reasons include missing CSI driver, insufficient permissions, or invalid parameters. Fix the specific cause before proceeding.

Once tested, migrate one workload at a time by changing its PVC to reference the new storage class. Do not bulk-edit all PVCs at once.

Verification and Diagnostics

Hardening is only as good as your ability to verify it. For storage classes, verification happens at three levels: the storage class itself, the CSI driver configuration, and the resulting volume.

Storage class level

kubectl get storageclass gp2-encrypted -o yaml

Look for the parameters section and verify the security-relevant keys. Expected snippet:

parameters:
  encrypted: "true"
  type: gp2

If encrypted is missing or set to "false", the volume is not encrypted. Recreate the storage class with the correct parameter.

CSI driver level

Some CSI drivers expose security controls through their own configuration, often stored in a ConfigMap or a secret. For the AWS EBS CSI driver, you can check the driver Pod logs for provisioning events:

kubectl logs -n kube-system deployment/ebs-csi-controller -c ebs-plugin | grep -i "encrypt"

Expected output may include lines like:

msg="Provisioning volume" volume_id="vol-0abcd1234..." encrypted=true

If the log shows encrypted=false, the driver ignored the parameter. Recheck the storage class and the driver version.

Volume level

After a PVC is bound, verify the actual cloud volume properties. For AWS, use the AWS CLI:

aws ec2 describe-volumes --volume-ids vol-0abcd1234... --query 'Volumes[0].Encrypted'

Expected output:

true

If it returns false, the storage class parameter did not take effect. Check the provisioner name and whether a different storage class was used due to default selection.

Diagnostics should always be read-only until you have confirmed the failure mode. Never delete a volume to “reset” it without first capturing its state and backup policy.

Failure Modes and Recovery

Storage class hardening has three common failure modes, each with a specific recovery path.

Failure 1: PVC stuck in Pending

This usually means the storage class cannot provision a volume. Possible causes: missing CSI driver, incorrect provisioner name, insufficient RBAC for the provisioner, or invalid parameters.

Diagnose:

kubectl describe pvc -n <namespace> <pvc-name>

Look for events like failed to provision volume with StorageClass "gp2-encrypted": rpc error: code = InvalidArgument desc = ....

Recovery:

• Verify the provisioner name matches the CSI driver exactly.
• Check that the CSI driver Pods are running in kube-system.
• Review the storage class parameters for typos.
• If the issue is RBAC, check the provisioner’s ServiceAccount and its ClusterRoleBindings.

Failure 2: Volume created but not accessible to Pod

Symptom: Pod fails to start with FailedMount or MountVolume.SetUp failed.

Diagnose:

kubectl describe pod <pod-name>

Look for events like MountVolume.SetUp failed for volume "pvc-..." : rpc error: code = Internal desc = ....

Recovery:

• Verify the volume is in the same availability zone as the node (for cloud providers).
• Check the CSI driver’s node plugin logs: kubectl logs -n kube-system <csi-node-pod> -c <driver-container>.
• If the volume was created with ReadWriteOnce, it can only be mounted on one node at a time. Ensure no other node is trying to mount it.

Failure 3: Security parameter silently ignored

This is the most dangerous failure because everything looks healthy. Example: you set encrypted: "true" but the cloud volume is not encrypted.

Diagnose as shown in the Verification section: compare storage class YAML, provisioner logs, and cloud volume attributes.

Recovery:

• If a parameter is ignored, check the CSI driver documentation. Some parameters require specific formats or driver versions.
• Test with a minimal storage class containing only that parameter. If it still fails, the driver may not support it.
• For existing volumes, you cannot retroactively enable encryption. You must create a new encrypted volume and migrate data. Plan a maintenance window.

Common Pitfalls and How to Avoid Them

Security hardening often fails because of these recurring mistakes.

Pitfall 1: Overly broad RBAC for storage classes

Developers often get * permissions on storage classes, PVCs, and PVs. This allows them to create arbitrary storage classes with dangerous parameters, delete volumes, or access other teams’ data.

How to avoid: Use least privilege. Define roles that only allow creating PVCs with specific storage classes.

Example Role for a developer namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev
  name: storage-user
rules:
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list"]

This allows developers to use existing storage classes but not create or modify them. Storage class creation should be an ops-only capability.

Pitfall 2: Hardcoding secrets in storage class parameters

Some storage classes accept credentials in the parameters field (e.g., for NFS or object storage). Storing secrets in plain text in the storage class YAML is a data leak waiting to happen.

How to avoid: Use Kubernetes Secrets and reference them through the CSI driver. Most CSI drivers support secretRef in the storage class or the PVC. For example, the Ceph CSI driver uses a secret for the provisioner, defined in a separate Secret and referenced by the storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-rbd
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: <cluster-id>
  pool: <pool-name>
  csi.storage.k8s.io/provisioner-secret-name: ceph-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system

The actual credentials live in ceph-secret, not in the storage class. Protect that Secret with RBAC and encryption at rest (e.g., use a KMS provider for etcd).

Pitfall 3: Ignoring volume binding modes

The volumeBindingMode field affects security. With Immediate binding, the volume is provisioned before any Pod uses it, potentially in the wrong availability zone or with incorrect topology. With WaitForFirstConsumer, the volume is not created until a Pod actually needs it, which allows the scheduler to place the volume correctly. However, WaitForFirstConsumer also means the volume is exposed only after a Pod binds to it, reducing the window for unauthorized access.

Check your storage class:

kubectl get storageclass gp2-encrypted -o jsonpath='{.volumeBindingMode}'

Expected output:

WaitForFirstConsumer

If it shows Immediate, consider whether you can switch. Changing binding mode on an existing storage class is not allowed; create a new class and migrate.

Pitfall 4: Neglecting CSI driver RBAC and configuration files

CSI drivers run as Pods with their own ServiceAccount and often need broad cluster permissions. Attackers who compromise a CSI driver can exfiltrate data. Harden CSI drivers by:

• Reviewing their RBAC permissions and removing any that are not documented as required.
• Running CSI driver containers as non-root users with runAsNonRoot: true.
• Restricting the driver’s network access to only necessary endpoints.
• Keeping the driver updated; security fixes are frequent.

Check the security context of the CSI driver Pod:

kubectl get pod -n kube-system -l app=ebs-csi-controller -o jsonpath='{.items[0].spec.containers[*].name}{"\n"}'
kubectl get pod -n kube-system -l app=ebs-csi-controller -o jsonpath='{.items[0].spec.securityContext}{"\n"}'

If the security context is empty or does not include runAsNonRoot: true, add it via a patch or update the Deployment.

Operations Checklist

Use this checklist to create a repeatable security review for storage classes. Each item has an owner and a review cadence.

Run the checklist in a read-only audit mode first. Only make changes after the audit confirms a gap.

Conclusion

Storage class security is a continuous practice, not a one-time configuration. By inventorying your environment, using safe configuration paths, verifying every change, and planning for failure modes, you reduce the risk of data exposure and misconfiguration. The examples in this article give you a starting point for hardening your own clusters.

As a next step, choose one low-risk verification from the checklist. Record the current state, run the documented command, compare the result with the expected signal, and review dependencies such as Persistent Volumes, Persistent Volume Claims, and CSI drivers. Then extend the process to the rest of your storage classes. A reliable workflow makes failure visible, protects sensitive values, and limits changes to the intended resource.