## Intro

Storage classes in Kubernetes define how persistent volumes are provisioned. They also carry security-sensitive settings: provisioner credentials, filesystem options, mount parameters, and the access modes that determine who can claim a volume. A misconfigured storage class can expose data, allow privilege escalation, or lock teams out of their own storage. This article walks through security hardening for storage classes with practical examples, commands, and verification steps. It is written for developers, DevOps engineers, and technical startup teams who manage Kubernetes clusters and need to move from “we saw a problem” to “we verified the fix.”

The goal is operational safety. We observe before changing, limit the blast radius of each change, use placeholders instead of real secrets, verify every result, and document recovery paths. Each section includes commands you can run in a test namespace, expected output, failure signals, and a recovery decision.

## Version and Environment Inventory

Before hardening anything, know your cluster version, the storage provisioner, and the CSI driver in use. Different Kubernetes versions and CSI drivers support different security features.

Run this read-only check first:

```bash
kubectl version --short
kubectl get storageclass
kubectl get csidriver
```

Expected output from `kubectl get storageclass` might look like:

```
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
gp2 (default)        kubernetes.io/aws-ebs   Delete          Immediate           true                   87d
fast-rbd             rbd.csi.ceph.com       Delete          Immediate           true                   12d
```

If your provisioner is a CSI driver, check its version and supported features with `kubectl describe csidriver <driver-name>`. For example, `kubectl describe csidriver fast-rbd` may show `attachable: true`, `podInfoOnMount: true`, and `volumeLifecycleModes: [Persistent]`. These flags affect how securely the driver can be configured.

The smallest change in this phase is often just documenting the environment. Create an inventory file in your ops repository:

```bash
# storage-inventory.md
Kubernetes version: 1.26.3
Provisioners: kubernetes.io/aws-ebs, rbd.csi.ceph.com
CSI drivers: ebs.csi.aws.com (v1.19.0), rbd.csi.ceph.com (v3.8.1)
Default storage class: gp2
```

Protect this inventory like any other internal document. It contains no secrets, but it tells an attacker exactly which storage drivers to target.

## Safe Configuration Path

The safest configuration path is: create a new storage class rather than modifying an existing one, test it in a dedicated namespace, then slowly migrate workloads. Never change the default storage class in place without a rollback plan.

Example: suppose you want to harden the `gp2` storage class to use encryption by default. Instead of editing the existing class, create a new one called `gp2-encrypted`:

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-encrypted
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
```

Save as `gp2-encrypted.yaml` and apply:

```bash
kubectl apply -f gp2-encrypted.yaml
kubectl get storageclass gp2-encrypted
```

Expected output:

```
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2-encrypted   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   5s
```

Now test it in a non-production namespace:

```bash
kubectl create namespace storage-test
kubectl apply -n storage-test -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-encrypted-pvc
spec:
  storageClassName: gp2-encrypted
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF
```

Check the PVC status:

```bash
kubectl get pvc -n storage-test test-encrypted-pvc
```

Expected output:

```
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
test-encrypted-pvc  Bound    pvc-1a2b3c4d-...                          1Gi        RWO            gp2-encrypted   10s
```

If the PVC remains `Pending`, describe it to see why:

```bash
kubectl describe pvc -n storage-test test-encrypted-pvc
```

Common reasons include missing CSI driver, insufficient permissions, or invalid parameters. Fix the specific cause before proceeding.

Once tested, migrate one workload at a time by changing its PVC to reference the new storage class. Do not bulk-edit all PVCs at once.

## Verification and Diagnostics

Hardening is only as good as your ability to verify it. For storage classes, verification happens at three levels: the storage class itself, the CSI driver configuration, and the resulting volume.

**Storage class level**

```bash
kubectl get storageclass gp2-encrypted -o yaml
```

Look for the `parameters` section and verify the security-relevant keys. Expected snippet:

```yaml
parameters:
  encrypted: "true"
  type: gp2
```

If `encrypted` is missing or set to `"false"`, the volume is not encrypted. Recreate the storage class with the correct parameter.

**CSI driver level**

Some CSI drivers expose security controls through their own configuration, often stored in a ConfigMap or a secret. For the AWS EBS CSI driver, you can check the driver Pod logs for provisioning events:

```bash
kubectl logs -n kube-system deployment/ebs-csi-controller -c ebs-plugin | grep -i "encrypt"
```

Expected output may include lines like:

```
msg="Provisioning volume" volume_id="vol-0abcd1234..." encrypted=true
```

If the log shows `encrypted=false`, the driver ignored the parameter. Recheck the storage class and the driver version.

**Volume level**

After a PVC is bound, verify the actual cloud volume properties. For AWS, use the AWS CLI:

```bash
aws ec2 describe-volumes --volume-ids vol-0abcd1234... --query 'Volumes[0].Encrypted'
```

Expected output:

```
true
```

If it returns `false`, the storage class parameter did not take effect. Check the provisioner name and whether a different storage class was used due to default selection.

Diagnostics should always be read-only until you have confirmed the failure mode. Never delete a volume to “reset” it without first capturing its state and backup policy.

## Failure Modes and Recovery

Storage class hardening has three common failure modes, each with a specific recovery path.

**Failure 1: PVC stuck in Pending**

This usually means the storage class cannot provision a volume. Possible causes: missing CSI driver, incorrect provisioner name, insufficient RBAC for the provisioner, or invalid parameters.

Diagnose:

```bash
kubectl describe pvc -n <namespace> <pvc-name>
```

Look for events like `failed to provision volume with StorageClass "gp2-encrypted": rpc error: code = InvalidArgument desc = ...`.

Recovery:

- Verify the provisioner name matches the CSI driver exactly.
- Check that the CSI driver Pods are running in `kube-system`.
- Review the storage class parameters for typos.
- If the issue is RBAC, check the provisioner’s ServiceAccount and its ClusterRoleBindings.

**Failure 2: Volume created but not accessible to Pod**

Symptom: Pod fails to start with `FailedMount` or `MountVolume.SetUp failed`.

Diagnose:

```bash
kubectl describe pod <pod-name>
```

Look for events like `MountVolume.SetUp failed for volume "pvc-..." : rpc error: code = Internal desc = ...`.

Recovery:

- Verify the volume is in the same availability zone as the node (for cloud providers).
- Check the CSI driver’s node plugin logs: `kubectl logs -n kube-system <csi-node-pod> -c <driver-container>`.
- If the volume was created with `ReadWriteOnce`, it can only be mounted on one node at a time. Ensure no other node is trying to mount it.

**Failure 3: Security parameter silently ignored**

This is the most dangerous failure because everything looks healthy. Example: you set `encrypted: "true"` but the cloud volume is not encrypted.

Diagnose as shown in the Verification section: compare storage class YAML, provisioner logs, and cloud volume attributes.

Recovery:

- If a parameter is ignored, check the CSI driver documentation. Some parameters require specific formats or driver versions.
- Test with a minimal storage class containing only that parameter. If it still fails, the driver may not support it.
- For existing volumes, you cannot retroactively enable encryption. You must create a new encrypted volume and migrate data. Plan a maintenance window.

## Common Pitfalls and How to Avoid Them

Security hardening often fails because of these recurring mistakes.

**Pitfall 1: Overly broad RBAC for storage classes**

Developers often get `*` permissions on storage classes, PVCs, and PVs. This allows them to create arbitrary storage classes with dangerous parameters, delete volumes, or access other teams’ data.

How to avoid: Use least privilege. Define roles that only allow creating PVCs with specific storage classes.

Example Role for a developer namespace:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev
  name: storage-user
rules:
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list"]
```

This allows developers to use existing storage classes but not create or modify them. Storage class creation should be an ops-only capability.

**Pitfall 2: Hardcoding secrets in storage class parameters**

Some storage classes accept credentials in the `parameters` field (e.g., for NFS or object storage). Storing secrets in plain text in the storage class YAML is a data leak waiting to happen.

How to avoid: Use Kubernetes Secrets and reference them through the CSI driver. Most CSI drivers support `secretRef` in the storage class or the PVC. For example, the Ceph CSI driver uses a secret for the provisioner, defined in a separate Secret and referenced by the storage class:

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-rbd
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: <cluster-id>
  pool: <pool-name>
  csi.storage.k8s.io/provisioner-secret-name: ceph-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
```

The actual credentials live in `ceph-secret`, not in the storage class. Protect that Secret with RBAC and encryption at rest (e.g., use a KMS provider for etcd).

**Pitfall 3: Ignoring volume binding modes**

The `volumeBindingMode` field affects security. With `Immediate` binding, the volume is provisioned before any Pod uses it, potentially in the wrong availability zone or with incorrect topology. With `WaitForFirstConsumer`, the volume is not created until a Pod actually needs it, which allows the scheduler to place the volume correctly. However, `WaitForFirstConsumer` also means the volume is exposed only after a Pod binds to it, reducing the window for unauthorized access.

Check your storage class:

```bash
kubectl get storageclass gp2-encrypted -o jsonpath='{.volumeBindingMode}'
```

Expected output:

```
WaitForFirstConsumer
```

If it shows `Immediate`, consider whether you can switch. Changing binding mode on an existing storage class is not allowed; create a new class and migrate.

**Pitfall 4: Neglecting CSI driver RBAC and configuration files**

CSI drivers run as Pods with their own ServiceAccount and often need broad cluster permissions. Attackers who compromise a CSI driver can exfiltrate data. Harden CSI drivers by:

- Reviewing their RBAC permissions and removing any that are not documented as required.
- Running CSI driver containers as non-root users with `runAsNonRoot: true`.
- Restricting the driver’s network access to only necessary endpoints.
- Keeping the driver updated; security fixes are frequent.

Check the security context of the CSI driver Pod:

```bash
kubectl get pod -n kube-system -l app=ebs-csi-controller -o jsonpath='{.items[0].spec.containers[*].name}{"\n"}'
kubectl get pod -n kube-system -l app=ebs-csi-controller -o jsonpath='{.items[0].spec.securityContext}{"\n"}'
```

If the security context is empty or does not include `runAsNonRoot: true`, add it via a patch or update the Deployment.

## Operations Checklist

Use this checklist to create a repeatable security review for storage classes. Each item has an owner and a review cadence.

| Checklist item | Owner (single accountable person) | Review frequency | Verification command or signal |
|---|---|---|---|
| Inventory all storage classes and provisioners | Priya Shah, Platform Lead | Monthly | `kubectl get storageclass` |
| Ensure default storage class meets encryption policy | Alex Chen, Security Engineer | Monthly | `kubectl get sc <default> -o yaml \| grep encrypted` |
| Review storage class RBAC for least privilege | Marcus Webb, DevOps | Quarterly | `kubectl auth can-i --list --as=system:serviceaccount:dev:developer` |
| Check CSI driver versions and security advisories | Priya Shah, Platform Lead | Weekly | `kubectl get csidriver -o yaml \| grep image` |
| Test a new storage class in a sandbox namespace before production | Alex Chen, Security Engineer | As needed | `kubectl apply -f new-sc.yaml -n sandbox; kubectl get pvc -n sandbox` |
| Verify encrypted volumes at cloud provider level | Marcus Webb, DevOps | Monthly | `aws ec2 describe-volumes --query 'Volumes[?Encrypted==`false`]'` |
| Rotate storage class secrets | Priya Shah, Platform Lead | Quarterly | `kubectl get secret -n kube-system \| grep csi` |

Run the checklist in a read-only audit mode first. Only make changes after the audit confirms a gap.

## Conclusion

Storage class security is a continuous practice, not a one-time configuration. By inventorying your environment, using safe configuration paths, verifying every change, and planning for failure modes, you reduce the risk of data exposure and misconfiguration. The examples in this article give you a starting point for hardening your own clusters.

As a next step, choose one low-risk verification from the checklist. Record the current state, run the documented command, compare the result with the expected signal, and review dependencies such as Persistent Volumes, Persistent Volume Claims, and CSI drivers. Then extend the process to the rest of your storage classes. A reliable workflow makes failure visible, protects sensitive values, and limits changes to the intended resource.