## Introduction Kubernetes authentication is the first gate every request must pass through before it reaches the API server. Whether you are a developer debugging a failing deployment, a platform engineer onboarding a new team, or an SRE investigating an unusual permission error, knowing the exact commands to inspect and verify authentication is essential for safe, efficient cluster operations. This article provides a practical command reference for Kubernetes authentication. It covers core concepts, common workflows, and diagnostic techniques, with concrete examples and expected outputs. We focus on commands that are safe to run in a production cluster, emphasize least privilege, and include recovery steps for common failure modes. The goal is operational clarity: understand who is authenticated, how they are authenticated, and whether their credentials are working as intended. You will learn to inspect certificates, tokens, service accounts, and validate Role-Based Access Control (RBAC) configurations with a few well-chosen commands. ## Version and Environment Inventory Before running authentication commands, know your cluster version and components. Authentication behavior can vary between Kubernetes versions, especially with features like TokenRequest API (stable since 1.20) and bound service account tokens (default since 1.21). Use the following read-only commands to gather information without changing state. ```bash kubectl version --short ``` Example output: ``` Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 ``` This confirms the client and server are compatible. If the server version is significantly older or newer than expected, some authentication features may not be available. Check the API server's authentication configuration by viewing its pod specification (if you have access to the control plane): ```bash kubectl -n kube-system get pod kube-apiserver- -o yaml | grep -A5 'command:' ``` Look for flags like `--authentication-token-webhook-config-file`, `--oidc-issuer-url`, or `--client-ca-file`. These indicate which authentication methods are enabled. ### Practical environment checks - Verify you can reach the API server and that your current credentials work: ```bash kubectl auth whoami ``` Output example: ``` ATTRIBUTE VALUE Username kubernetes-admin UID abc123... (on some clusters) Groups [system:masters system:authenticated] ``` If you get `Error from server (Forbidden)`, your user lacks the `selfsubjectaccessreviews` permission, but basic authentication still works. - List nodes to confirm cluster health: ```bash kubectl get nodes ``` A `NotReady` node may indicate certificate or kubelet authentication issues. - Check the default namespace's service accounts: ```bash kubectl get serviceaccounts ``` Each namespace has a `default` service account. Problems with this account often cause authentication failures for workloads. Keep all commands read-only until you have established a baseline. Use `kubectl` with `--dry-run=client` for any change to understand what resources would be modified. ## Understanding Kubernetes Authentication Kubernetes does not manage user accounts natively. Instead, it relies on external identity providers (X.509 certificates, static tokens, OpenID Connect, webhooks) for human users, and service accounts for processes running inside the cluster. All authenticated requests are then authorized separately, usually via RBAC. ### Core commands for authentication inspection #### 1. Inspect your current credentials Your kubeconfig file stores cluster, user, and context information. Use `kubectl config view` to see it: ```bash kubectl config view --minify ``` This shows the current context, cluster, and user details. Example: ```yaml apiVersion: v1 clusters: - cluster: certificate-authority-data: REDACTED server: https://192.168.1.100:6443 name: my-cluster contexts: - context: cluster: my-cluster user: my-user name: my-context current-context: my-context kind: Config preferences: {} users: - name: my-user user: client-certificate-data: REDACTED client-key-data: REDACTED ``` To see the decoded certificate details (expiry, subject, issuer): ```bash kubectl config view --raw -o jsonpath='{.users[?(@.name=="my-user")].user.client-certificate-data}' | base64 -d | openssl x509 -noout -subject -issuer -dates ``` Output may look like: ``` subject=O=system:masters, CN=kubernetes-admin issuer=CN=kubernetes notBefore=Jan 1 00:00:00 2023 GMT notAfter=Dec 31 23:59:59 2024 GMT ``` Check the expiration date. An expired certificate causes authentication failures. #### 2. Check service account tokens Each service account is associated with one or more secrets containing tokens. List them: ```bash kubectl get secrets ``` Look for secrets of type `kubernetes.io/service-account-token`. For a specific service account `default`: ```bash kubectl get secret $(kubectl get serviceaccount default -o jsonpath='{.secrets[0].name}') -o yaml ``` The token is base64-encoded. Decode it to inspect the JWT: ```bash kubectl get secret -o jsonpath='{.data.token}' | base64 -d ``` This will show the JWT. You can decode the payload to see claims: ```bash # Copy the token and paste it into jwt.io, or use a command-line tool ``` In Kubernetes 1.24+, legacy service account token secrets are no longer automatically created. Use TokenRequest API to generate a short-lived token: ```bash kubectl create token default ``` This produces a token valid for 1 hour by default. Example output: ``` eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjg4NzY1NDMyLCJpYXQiOjE2ODg3NjE4MzIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VydmljZWFjY291bnQiOiJ7XCJuYW1lXCI6XCJkZWZhdWx0XCIsXCJ1aWQiXCI6XCI5ZDc5YzA1MS05YjQxLTQ0MjEtOGVjMC0wM2ExMjI2OTk2YmZcIn0ifSwibmJmIjoxNjg4NzYxODMyLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWZhdWx0In0.eyJhbGciOiJSUzI1NiJ9... (truncated) ``` #### 3. Verify RBAC permissions Use `kubectl auth can-i` to check if a specific action is allowed for the current user or a service account. Check if you can create pods in the current namespace: ```bash kubectl auth can-i create pods ``` Output: `yes` or `no`. For a service account `my-sa` in namespace `my-ns`, check if it can list deployments: ```bash kubectl auth can-i list deployments --as=system:serviceaccount:my-ns:my-sa -n my-ns ``` To see who can perform an action across the cluster, use `kubectl auth reconcile` but that is for applying RBAC changes. For troubleshooting, inspect roles and bindings. ## Common Authentication Workflows with Examples ### Adding a new user with client certificates 1. Generate a private key and certificate signing request (CSR): ```bash openssl genrsa -out user.key 2048 openssl req -new -key user.key -out user.csr -subj "/CN=alice/O=developers" ``` The `CN` is the username, and `O` becomes the group. 2. Create a Kubernetes CertificateSigningRequest resource: ```bash cat < user.crt ``` 5. Add the user to kubeconfig: ```bash kubectl config set-credentials alice --client-certificate=user.crt --client-key=user.key kubectl config set-context alice-context --cluster=my-cluster --user=alice kubectl config use-context alice-context ``` 6. Test access: ```bash kubectl get pods ``` If no RBAC permissions are assigned, output will be `Error from server (Forbidden): pods is forbidden: User "alice" cannot list resource "pods" in API group "" in the namespace "default"`. 7. Grant permissions using a Role and RoleBinding: ```bash kubectl create role pod-reader --verb=get,list,watch --resource=pods kubectl create rolebinding alice-pod-reader --role=pod-reader --user=alice ``` Now `kubectl get pods` should succeed. ### Creating a service account for an application 1. Create a dedicated service account: ```bash kubectl create serviceaccount my-app-sa ``` 2. Assign it to a deployment: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 1 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: serviceAccountName: my-app-sa containers: - name: app image: nginx ``` 3. Verify the pod is using the service account: ```bash kubectl get pod -l app=my-app -o jsonpath='{.items[0].spec.serviceAccountName}' ``` Output: `my-app-sa`. 4. Grant the service account access to needed resources. For example, allow it to read configmaps in its namespace: ```bash kubectl create role configmap-reader --verb=get,list --resource=configmaps kubectl create rolebinding my-app-sa-configmap-reader --role=configmap-reader --serviceaccount=default:my-app-sa ``` ### Using OpenID Connect (OIDC) authentication If your cluster is configured with OIDC, users obtain an ID token from the identity provider and use it as a bearer token. The API server validates the token against the provider's configuration. To use an OIDC token, provide it directly in the kubeconfig: ```bash kubectl config set-credentials oidc-user --auth-provider=oidc --auth-provider-arg=idp-issuer-url=https://issuer.example.com --auth-provider-arg=client-id=kubernetes --auth-provider-arg=id-token= ``` Or you can pass the token with each request: ```bash kubectl --token= get pods ``` To inspect the token claims, decode the JWT payload. The `sub` claim becomes the username, and `groups` claim becomes group memberships. ## Verification and Diagnostics ### Diagnosing authentication failures Authentication failures typically result in HTTP 401 errors from the API server. Common messages: - `Unauthorized` - `error: You must be logged in to the server (Unauthorized)` - `x509: certificate signed by unknown authority` Use `kubectl` with increased verbosity to see detailed request/response: ```bash kubectl get pods -v=8 ``` Look for lines containing `401` or `Authorization`. If using a token, check its expiration and signature: ```bash kubectl create token default --duration=0s # In older versions, may fail; use manual decode. ``` Decode the token with a tool like `jq` and `base64`: ```bash # Extract payload (second segment) TOKEN=$(kubectl create token default) echo $TOKEN | cut -d'.' -f2 | base64 -d 2>/dev/null | jq . ``` Example output: ```json { "aud": ["https://kubernetes.default.svc.cluster.local"], "exp": 1688765432, "iat": 1688761832, "iss": "https://kubernetes.default.svc.cluster.local", "kubernetes.io": { "namespace": "default", "serviceaccount": {"name":"default","uid":"9d79c051-..."} }, "nbf": 1688761832, "sub": "system:serviceaccount:default:default" } ``` Check that `exp` is in the future and `aud` includes the API server audience. ### Verifying RBAC policies To see which roles are bound to a user or service account: ```bash kubectl get rolebindings,clusterrolebindings --all-namespaces -o custom-columns='KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name' | grep alice ``` For a deeper analysis, use `kubectl auth reconcile --dry-run=client -f rbac.yaml` to see differences without applying. ### Checking service account token mounts Inside a pod, the service account token is mounted at `/var/run/secrets/kubernetes.io/serviceaccount/token`. To verify the token is the expected one: ```bash kubectl exec -- cat /var/run/secrets/kubernetes.io/serviceaccount/token ``` Compare with the token from the secret or TokenRequest. ## Failure Modes and Recovery Understanding common authentication failures helps you respond quickly. ### Expired client certificate **Symptom**: When running `kubectl get pods`, you see: ``` Unable to connect to the server: x509: certificate has expired or is not yet valid ``` **Why it happens**: The client certificate in kubeconfig has passed its `notAfter` date. **Recovery**: 1. Check expiry with `openssl x509 -noout -dates -in `. 2. Obtain a new certificate from your CA or regenerate via CSR as described earlier. 3. Update kubeconfig credentials. ### Invalid or expired token **Symptom**: API requests return `401 Unauthorized: invalid token`. **Why it happens**: The token may be expired, malformed, or the signing key has rotated. **Recovery**: - For service account tokens: generate a new token with `kubectl create token ` and update your application. - For static tokens: check the API server's `--token-auth-file` and ensure the token matches. - For OIDC: refresh the ID token with the provider. ### Certificate signed by unknown authority **Symptom**: `x509: certificate signed by unknown authority` when connecting. **Why it happens**: The cluster's CA certificate in kubeconfig does not match the server's certificate, or the CA is not trusted. **Recovery**: - Ensure the `certificate-authority-data` or `certificate-authority` in kubeconfig points to the correct CA. - If using a custom CA, add it to your system trust store or specify `insecure-skip-tls-verify: true` (not recommended for production). ### User cannot perform action despite role binding **Symptom**: `kubectl auth can-i create pods` returns `no` but you believe the user should have permission. **Why it happens**: RBAC is misconfigured: role binding in wrong namespace, wrong subject (user vs service account), or missing verbs/resources. **Recovery**: - List bindings with `kubectl get rolebindings -n -o yaml` and verify subjects. - Check the role's rules: `kubectl get role -o yaml`. - Use `kubectl auth can-i --list --as=` to see all permissions for that user. ### Service account not mounting token **Symptom**: Application inside pod fails to authenticate to API server, and the token file is missing or empty. **Why it happens**: The pod's service account name may be misspelled, or the service account does not exist. **Recovery**: - Verify service account: `kubectl get serviceaccounts -n `. - Check pod spec: `kubectl get pod -o jsonpath='{.spec.serviceAccountName}'`. - If missing, recreate pod with correct `serviceAccountName`. ## Operations Checklist Use this checklist to ensure authentication is properly managed in your cluster. Assign an owner and frequency for review. | Item | Command / Action | Owner | Frequency | |------|-------------------|-------|-----------| | Audit active users and their groups | `kubectl config view --raw` review | Security Engineer (e.g., Priya Shah) | Monthly | | Review service accounts and their tokens | `kubectl get serviceaccounts --all-namespaces` and inspect secrets | Platform Lead (e.g., Marcus Chen) | Monthly | | Check for expired certificates | `kubectl get csr`, `openssl x509 -noout -dates` on all client certs | Cluster Admin (e.g., Alice Johnson) | Weekly | | Validate RBAC policies against least privilege | `kubectl auth reconcile --dry-run` and `kubectl auth can-i --list` for critical users | Security Officer (e.g., David Kim) | Quarterly | | Rotate service account tokens | `kubectl create token ` for short-lived tokens | DevOps Engineer (e.g., Sarah Lee) | Quarterly | | Review authentication webhook configuration | Inspect API server flags | Platform Architect (e.g., Tom Brown) | Semi-annually | | Test incident response for authentication failure | Simulate expired cert or token and run recovery procedure | SRE Team (single owner: Emily Davis) | Annually | ### Key operational practices 1. **Use RBAC with least privilege**: Grant only necessary permissions. Avoid using `cluster-admin` for daily work. 2. **Prefer short-lived tokens**: For service accounts, use TokenRequest API or projected tokens with expiration. For users, use OIDC with refresh tokens. 3. **Monitor authentication metrics**: Enable API server audit logs and alert on `401` surges. 4. **Automate certificate renewal**: Use cert-manager or a similar controller for client certificates. 5. **Document recovery steps**: Maintain runbooks for expired certificates, token issues, and RBAC misconfigurations. ## Common Pitfalls and How to Avoid Them ### Pitfall 1: Using the default service account for all workloads **Why it happens**: It is convenient because every namespace has one, and developers may not think about identity isolation. **How to avoid**: Create a dedicated service account for each application and explicitly set `serviceAccountName` in pod specs. This limits blast radius if a token is compromised. **Recovery**: If already using default, create new service accounts, update deployments, and delete default token mounts with `automountServiceAccountToken: false` where not needed. ### Pitfall 2: Overly broad RBAC permissions with wildcards **Why it happens**: Developers often request `*` on resources to get things working quickly, or admins copy cluster-admin role for simplicity. **How to avoid**: Start with minimal permissions, test with `kubectl auth can-i --list`, and expand only when necessary. Use tools like `kubectl-who-can` or `audit2rbac` to generate least-privilege roles. **Recovery**: Review all role bindings for wildcard usage with `kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name=="cluster-admin")'` and tighten. ### Pitfall 3: Hardcoding service account tokens in application code or config files **Why it happens**: It seems easier than mounting from secrets, especially when running outside Kubernetes. **How to avoid**: Use the mounted token file or TokenRequest API within the cluster. Outside, use workload identity or OIDC. **Recovery**: Remove hardcoded tokens, rotate them immediately, and adopt secure token delivery. ### Pitfall 4: Ignoring certificate expiry **Why it happens**: Client certificates have long lifetimes, and no alerting is set up. **How to avoid**: Monitor certificate expiry with Prometheus alerts or cert-manager notifications. **Recovery**: Implement automated renewal and have a manual process as backup. ### Pitfall 5: Not differentiating between authentication and authorization **Why it happens**: Users often confuse the two: they see `Forbidden` (403) and think authentication is broken. **How to avoid**: Diagnose with `kubectl auth can-i`. If `whoami` succeeds but `can-i` returns `no`, it is an authorization (RBAC) issue, not authentication. **Recovery**: Fix the role binding rather than trying to change user credentials. ### Pitfall 6: Running with cluster-admin for CI/CD pipelines **Why it happens**: CI/CD needs broad access to deploy across namespaces, and granting cluster-admin is easy. **How to avoid**: Create dedicated service accounts with narrowly scoped roles for each pipeline. Use Kubernetes namespace isolation. **Recovery**: Audit pipeline credentials, create new roles, and rotate tokens. ## Conclusion Authentication is the foundation of Kubernetes security. Mastering the commands to inspect and manage authentication ensures that you can quickly resolve access issues, maintain least privilege, and keep your cluster secure. This article provided a comprehensive reference: from checking your environment and credentials, to creating users and service accounts, to diagnosing failures and avoiding common pitfalls. Start by applying one low-risk verification from this guide: run `kubectl auth whoami` to see who you are authenticated as, then check your permissions with `kubectl auth can-i --list`. Document the results and compare them with expected access. Then review your service accounts and token practices. Over time, expand to automated checks and audits. A robust authentication operation system makes failures visible, protects sensitive credentials, limits changes to intended resources, and ensures recovery plans are tested before an incident occurs.