## Intro

TLS certificate upgrades and migrations can cause avoidable outages if done ad hoc. A missed SAN, an unexpected client that cannot negotiate ECDSA, or a chain change applied at peak time are common failure modes.

This guide gives you a concrete, low-risk path: plan the change, prove it locally, roll out with a canary, validate behavior, and keep rollback one command away. You will see working examples for Nginx and Kubernetes Ingress on Linux so you can practice before production.

## Workflow Overview

Use this checklist to keep risk low and progress visible.

- Inventory TLS termination

- Reverse proxies: Nginx, Envoy, HAProxy

- Kubernetes: Ingress controllers and gateways

- Managed load balancers and any direct TLS listeners

- Map each to where certs and keys live and how they reload

- Example : run openssl s_client -connect host:443 -servername host </dev/null to capture the current certificate details for each endpoint.

- Decide the upgrade scope

- Key algorithm: RSA 2048 → RSA 3072, or move to ECDSA P-256

- Signature and chain: new intermediate or root, chain file format

- SAN and hostname changes: add or remove names, wildcard vs specific SANs

- Certificate lifetime and renewal flow

- Backward compatibility: consider dual certs (ECDSA + RSA) to cover old clients

- Prepare artifacts

- Generate keys and CSRs on a hardened host

- Plan storage: file permissions, auditing, and backups for private keys

- For dual stack, produce both RSA and ECDSA artifacts

- Stage safely

- Load new certs into a non-production proxy or canary host

- Confirm supported protocols (TLS 1.2 and 1.3) and disable older ones

- Verify the full chain and OCSP stapling behavior if enabled

- Deployment plan

- Prefer atomic switches: symlinks for files, versioned secrets for Kubernetes

- Use a canary hostname or path to exercise the new certificate with real traffic

- Reload gracefully only after validation passes

- Validate thoroughly

- Hostname, SANs, signature algorithm, chain order

- Protocol negotiation: TLS 1.2 and 1.3

- Cipher negotiation for both ECDSA and RSA where applicable

- Application behavior and error rates

- Observe

- Watch proxy and controller logs for handshake or certificate errors

- Track 4xx/5xx rates and client error reports during the window

- Rollback ready

- Keep the previous cert and key installed but not active

- Revert by flipping a symlink or switching a Kubernetes secret reference

- Reload the proxy and re-validate

- Clean up and document

- Remove old material after a safe window

- Capture commands, file paths, and checks in a runbook

## Local Pilot Plan

Start with a single hostname or route that you can exercise safely. Prove the following before production:

- You can install the new certificate alongside the old one

- You can validate chain, protocols, and cipher suites locally

- You can switch and roll back in under a minute

### Generate keys and CSRs (Linux, OpenSSL)

ECDSA P-256 :

openssl ecparam -name prime256v1 -genkey -noout -out server-ecdsa.key
openssl req -new -key server-ecdsa.key -out server-ecdsa.csr \
 -subj '/CN=example.com' \
 -addext 'subjectAltName=DNS:example.com, DNS:www.example.com' 
 RSA 3072 :

openssl genrsa -out server-rsa.key 3072
openssl req -new -key server-rsa.key -out server-rsa.csr \
 -subj '/CN=example.com' \
 -addext 'subjectAltName=DNS:example.com, DNS:www.example.com' 
 When issued, you should have private keys and full chain files, for example:

- server-ecdsa.key and example-ecdsa-fullchain.pem

- server-rsa.key and example-rsa-fullchain.pem

Set strict permissions on private keys:

sudo chown root:root server-*.key
sudo chmod 600 server-*.key 

### Nginx reverse proxy: dual certificate canary

 Place certs and keys under a versioned path and point stable symlinks to the active release:

sudo install -d -m 0750 /etc/nginx/certs/releases/2024-07-20
sudo cp example-ecdsa-fullchain.pem /etc/nginx/certs/releases/2024-07-20/
sudo cp server-ecdsa.key /etc/nginx/certs/releases/2024-07-20/
sudo cp example-rsa-fullchain.pem /etc/nginx/certs/releases/2024-07-20/
sudo cp server-rsa.key /etc/nginx/certs/releases/2024-07-20/

sudo ln -sfn /etc/nginx/certs/releases/2024-07-20/example-ecdsa-fullchain.pem /etc/nginx/certs/example-ecdsa-fullchain.pem
sudo ln -sfn /etc/nginx/certs/releases/2024-07-20/server-ecdsa.key /etc/nginx/certs/example-ecdsa.key
sudo ln -sfn /etc/nginx/certs/releases/2024-07-20/example-rsa-fullchain.pem /etc/nginx/certs/example-rsa-fullchain.pem
sudo ln -sfn /etc/nginx/certs/releases/2024-07-20/server-rsa.key /etc/nginx/certs/example-rsa.key 
 Configure Nginx to offer both ECDSA and RSA, and restrict protocols:

server {
 listen 443 ssl http2;
 server_name example.com www.example.com;

 ssl_protocols TLSv1.2 TLSv1.3;
 ssl_ciphers HIGH:!aNULL:!MD5;
 ssl_prefer_server_ciphers off;
 ssl_ecdh_curve X25519:secp256r1;

 # ECDSA cert
 ssl_certificate /etc/nginx/certs/example-ecdsa-fullchain.pem;
 ssl_certificate_key /etc/nginx/certs/example-ecdsa.key;

 # RSA cert
 ssl_certificate /etc/nginx/certs/example-rsa-fullchain.pem;
 ssl_certificate_key /etc/nginx/certs/example-rsa.key;

 location / {
 proxy_pass http://app:8080;
 proxy_set_header Host $host;
 proxy_set_header X-Forwarded-Proto https;
 }
} 
 Test and reload:

sudo nginx -t && sudo systemctl reload nginx 
 Validate negotiation and chain locally:

# Check TLS 1.3
openssl s_client -connect example.com:443 -servername example.com -tls1_3 -brief </dev/null

# Force ECDSA cipher to confirm ECDSA path
openssl s_client -connect example.com:443 -servername example.com \
 -cipher ECDHE-ECDSA-AES128-GCM-SHA256 </dev/null

# Force RSA cipher to confirm RSA fallback
openssl s_client -connect example.com:443 -servername example.com \
 -cipher ECDHE-RSA-AES128-GCM-SHA256 </dev/null

# Curl with host-to-IP pin for previews
curl -v --resolve example.com:443:203.0.113.10 https://example.com/ 
 Rollback plan for Nginx:

- Keep previous certs under an older release path

- Flip symlinks back and reload if needed

sudo ln -sfn /etc/nginx/certs/releases/2024-06-01/example-ecdsa-fullchain.pem /etc/nginx/certs/example-ecdsa-fullchain.pem
sudo ln -sfn /etc/nginx/certs/releases/2024-06-01/server-ecdsa.key /etc/nginx/certs/example-ecdsa.key
sudo ln -sfn /etc/nginx/certs/releases/2024-06-01/example-rsa-fullchain.pem /etc/nginx/certs/example-rsa-fullchain.pem
sudo ln -sfn /etc/nginx/certs/releases/2024-06-01/server-rsa.key /etc/nginx/certs/example-rsa.key
sudo nginx -t && sudo systemctl reload nginx 

### Kubernetes Ingress: canary then switch

 Create a new secret for the canary hostname:

kubectl -n web create secret tls web-tls-new \
 --cert=example-fullchain.pem --key=server.key 
 Create a canary Ingress using the new secret and a separate host:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: web-canary
 namespace: web
spec:
 ingressClassName: nginx
 tls:
 - hosts:
 - canary.example.com
 secretName: web-tls-new
 rules:
 - host: canary.example.com
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: web
 port:
 number: 80 
 Apply and test:

kubectl apply -f web-canary.yaml
curl -v https://canary.example.com/
openssl s_client -connect canary.example.com:443 -servername canary.example.com -brief </dev/null 
 Switch production Ingress to the new secret:

- Ensure you keep the old secret, for example web-tls-old

- Update the production Ingress spec.tls.secretName to web-tls-new

# excerpt of production Ingress
spec:
 tls:
 - hosts:
 - example.com
 - www.example.com
 secretName: web-tls-new 
 Apply and watch the controller logs for updates:

kubectl -n web apply -f web-ingress.yaml
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller -f | grep -i tls 
 Rollback plan for Kubernetes:

- Revert secretName back to web-tls-old and apply

- Keep both secrets during the rollback window

## Validation checklist

- Hostname and SANs match what clients use

- Protocols: TLS 1.2 and 1.3 enabled, older disabled

- Cipher negotiation: ECDSA path available, RSA fallback works if needed

- Chain is complete and ordered correctly

- No handshake or certificate errors in logs

- Traffic and error rates normal after cutover

## Observability and safety notes

- Nginx: tail error.log for SSL_do_handshake errors during the window

- Ingress controller: watch logs for secret updates and handshake issues

- Keep a short maintenance window with on-call and clear rollback triggers

## Conclusion

Successful TLS upgrades follow a repeatable shape: inventory, prepare artifacts, canary the change, validate aggressively, and keep rollback one command away. Start with a narrow pilot that you can inspect locally, prove negotiation and chain integrity, then scale the pattern to more services. Document the exact commands, file paths, and checks, and schedule renewals well before expiry so future rotations are routine.