Intro
TLS certificate errors are a leading cause of unexpected downtime, browser warnings, and failed API calls. This guide helps developers, DevOps engineers, and technical startup teams move from observed symptoms to verified fixes. We focus on practical diagnostics using OpenSSL, configuration examples for common servers like Nginx, and Kubernetes Ingress specifics. Every recommendation includes a concrete command, expected output, and a way to confirm the fix. The goal is operational safety: observe before changing, limit blast radius, protect secrets, and verify outcomes.
You will learn how to inspect certificates locally and remotely, validate chains, identify common error patterns, and implement fixes with rollback in mind. While we emphasize Nginx, Kubernetes Ingress, and Linux, the principles apply broadly.
Version and Environment Inventory
Before troubleshooting, establish a clear picture of your environment. Record the operating system and version, OpenSSL version, web server or reverse proxy version, and Kubernetes version if applicable. This information is crucial because commands and configuration syntax can vary between versions.
Example inventory commands:
# Check OpenSSL version
openssl version
# Expected output: OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
# Check Nginx version
nginx -v
# Expected output: nginx version: nginx/1.24.0
# Check Kubernetes version (if used)
kubectl version --short
Verify that you have read access to certificate files and the necessary permissions to reload services. Confirm the certificate file path and private key path configured in your server settings. Document the certificate's location, issuing CA, and intended hostnames.
Inspecting a Local Certificate File
To inspect a certificate without modifying it, use openssl x509:
openssl x509 -in /etc/ssl/certs/example.com.pem -noout -subject -issuer -serial -dates -fingerprint -sha256
Expected output includes:
- subject: the certificate's common name and organization
- issuer: the certificate authority that signed it
- serial: unique serial number
- dates: validity period (notBefore and notAfter)
- SHA256 fingerprint: for pinning and verification
Record these values. A mismatch in subject or issuer often points to the wrong certificate file or incomplete chain.
Testing a Remote Endpoint
To check the certificate served by a live website, use openssl s_client:
echo | openssl s_client -connect example.com:443 -servername example.com -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -dates
This command connects and retrieves the certificate chain. Check that the hostname matches, the chain includes the necessary intermediate, and the certificate is within its validity period. A successful TCP connection alone does not guarantee a valid certificate.
Validating a Certificate Chain Offline
To verify that a certificate chain is trusted, use openssl verify:
openssl verify -CAfile trusted-ca.pem -untrusted intermediate.pem leaf.pem
A clean result is leaf.pem: OK. Errors such as unable to get local issuer certificate indicate a missing intermediate or root in the trust store.
Always use placeholders in documentation and protect private keys. Test renewal and rollback procedures before expiry becomes urgent.
Safe Configuration Path
Configuration errors are common when adding or updating TLS certificates. Always make changes in a controlled manner: back up current configuration, apply changes to a test environment first, and have a rollback plan.
Nginx Configuration Example
A minimal Nginx server block for TLS might look like:
server {
listen 443 ssl;
server_name example.com;
ssl_certificate /etc/ssl/certs/example.com.pem;
ssl_certificate_key /etc/ssl/private/example.com.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
}
After editing, test the configuration before reloading:
nginx -t
# Expected output: nginx: configuration file /etc/nginx/nginx.conf test is successful
If the test passes, reload Nginx:
systemctl reload nginx
Then verify the live certificate with openssl s_client as shown earlier.
Kubernetes Ingress Example
For Kubernetes, certificates are typically managed as Secrets. Create or update a TLS secret:
kubectl create secret tls example-tls --cert=path/to/tls.crt --key=path/to/tls.key -n default
Then reference it in an Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
spec:
tls:
- hosts:
- example.com
secretName: example-tls
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: example-service
port:
number: 80
Apply the changes and check that the Ingress is using the correct certificate:
kubectl apply -f ingress.yaml
kubectl describe ingress example-ingress
Always verify that the certificate is valid and the private key matches using openssl x509 -noout -modulus and openssl rsa -noout -modulus and comparing outputs.
Verification and Diagnostics
After any change, verify that the certificate is correctly installed and served. In addition to openssl s_client, use browser developer tools or online SSL checkers for a user perspective.
Checking Certificate Chain on a Remote Server
To see the entire chain, use:
echo | openssl s_client -connect example.com:443 -servername example.com -showcerts 2>/dev/null | grep -E "s:|i:"
This prints the subject and issuer of each certificate in the chain. Ensure that the chain ends with a trusted root.
Diagnosing Specific Errors with openssl s_client
The output of openssl s_client includes a verification result line such as:
Verify return code: 0 (ok)– certificate is validVerify return code: 18 (self signed certificate)– the certificate is self-signed and not trustedVerify return code: 20 (unable to get local issuer certificate)– intermediate missing from server configurationVerify return code: 21 (unable to verify the first certificate)– the first certificate in the chain is not the server certificate or the chain is incompleteVerify return code: 10 (certificate has expired)– certificate is expiredVerify return code: 62 (hostname mismatch)– certificate CN/SAN does not match the requested hostname
Use these codes to quickly identify the root cause.
Failure Modes and Recovery
Understanding common failure modes helps you prepare and respond quickly. Below are frequent TLS certificate issues and how to recover from each.
1. Expired Certificate
Symptom: Browsers show "Your connection is not private" with error code NET::ERR_CERT_DATE_INVALID. openssl s_client returns verify code 10.
Recovery: Renew the certificate with your CA, replace the certificate file and key, and reload the service. Set up monitoring to alert before expiry. For automated renewal, consider using Let's Encrypt with Certbot or a Kubernetes cert-manager.
2. Certificate Not Trusted
Symptom: Browser shows NET::ERR_CERT_AUTHORITY_INVALID. openssl s_client shows Verify return code: 19 (self signed certificate in certificate chain).
Recovery: Ensure you are using a certificate from a publicly trusted CA for public-facing sites. For internal sites, install your organization's root CA on client devices. On the server, include the correct intermediate certificate(s) in the chain.
3. Hostname Mismatch
Symptom: Browser shows NET::ERR_CERT_COMMON_NAME_INVALID. The certificate is valid for a different domain.
Recovery: Obtain a certificate with the correct Common Name (CN) or Subject Alternative Name (SAN). Wildcard certificates (*.example.com) can cover subdomains but not the root domain. Ensure the server is configured to serve the correct certificate for the requested hostname.
4. Incomplete Certificate Chain
Symptom: Browser may still work if it has cached intermediates, but some clients fail. openssl s_client may return code 21 or show missing issuer.
Recovery: Concatenate the server certificate with the intermediate(s) in the correct order (server cert first, then intermediates) and configure that file as the ssl_certificate. Use openssl verify -CAfile root.pem -untrusted intermediate.pem server.pem to validate offline.
5. Private Key Mismatch
Symptom: Web server fails to start and logs show "key values mismatch" or similar. Nginx error: SSL_CTX_use_PrivateKey_file: key values mismatch.
Recovery: Verify that the private key corresponds to the certificate by comparing their public keys:
openssl x509 -noout -modulus -in certificate.pem | openssl sha256
openssl rsa -noout -modulus -in private.key | openssl sha256
If outputs differ, find the correct key or generate a new certificate/key pair.
Common Pitfalls and How to Avoid Them
Even experienced engineers make avoidable mistakes. Here are some common pitfalls and prevention strategies.
Pitfall 1: Not Setting Up Monitoring for Expiry
Why it happens: Overlooked during initial setup; small teams with many certificates lose track.
How to avoid: Use monitoring tools like Nagios, Zabbix, or a simple cron job that checks expiry using openssl x509 -checkend and sends alerts. For Kubernetes, use cert-manager with automatic renewal.
Pitfall 2: Storing Private Keys in Version Control
Why it happens: Convenience or lack of awareness.
How to avoid: Never commit private keys to Git. Use secret management tools like HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets. If a key is exposed, revoke the certificate and issue a new one.
Pitfall 3: Not Including Intermediate Certificates
Why it happens: Misunderstanding of certificate chain requirements.
How to avoid: Always concatenate the full chain, test with openssl verify and online SSL checkers before deploying. Document the chain building process.
Pitfall 4: Ignoring Certificate Revocation
Why it happens: Rarely encountered unless a security incident occurs.
How to avoid: Understand how your CA handles revocation (CRL, OCSP). For critical services, enable OCSP stapling in Nginx: ssl_stapling on; ssl_stapling_verify on; and specify ssl_trusted_certificate.
Pitfall 5: Using Weak Cipher Suites or Protocols
Why it happens: Legacy configurations or outdated templates.
How to avoid: Use modern configurations from trusted sources like Mozilla SSL Configuration Generator. Regularly update TLS versions and disable deprecated protocols like TLSv1.0 and TLSv1.1.
Operations Checklist
Use this checklist to ensure a safe and thorough TLS certificate deployment or fix. Assign a single owner for each item, and review monthly.
| # | Action | Owner | Verification Command | Expected Result | Frequency |
|---|---|---|---|---|---|
| 1 | Document certificate inventory and expiry dates | DevOps Engineer (Priya Shah) | openssl x509 -in cert.pem -noout -dates | List of all certs with notAfter dates | Monthly |
| 2 | Verify certificate chain offline before deployment | Systems Administrator (Carlos Gomez) | openssl verify -CAfile ca.pem -untrusted intermediate.pem cert.pem | cert.pem: OK | Every change |
| 3 | Test configuration before reloading service | DevOps Engineer (Priya Shah) | nginx -t | syntax is ok and test is successful | Every change |
| 4 | Check live certificate after deployment | Security Engineer (Amelia Chen) | echo | openssl s_client -connect host:443 -servername host 2>/dev/null | openssl x509 -noout -dates -subject | Correct subject and valid dates | Every change |
| 5 | Update monitoring alert thresholds | Site Reliability Engineer (David Kim) | Review monitoring dashboard | Alerts trigger at 30 days before expiry | Monthly |
| 6 | Rotate private keys and certificates | DevOps Engineer (Priya Shah) | kubectl create secret tls ... or copy new files | New cert served, old key revoked | Annually or per policy |
| 7 | Review OCSP stapling configuration | Security Engineer (Amelia Chen) | openssl s_client -connect host:443 -status | OCSP response: no error | Quarterly |
For each major change, write a rollback plan. For example: if the new certificate causes errors, restore the previous certificate file and reload the service. Store backups of previous certificates and private keys securely, with restricted access.
Conclusion
TLS certificate errors can disrupt service and erode trust. A methodical approach—inspect, diagnose, change, verify, and document—reduces downtime and prevents recurrence. With the OpenSSL commands and configuration examples in this guide, you can confidently troubleshoot common issues in Nginx, Kubernetes Ingress, and general Linux environments. Remember to assign clear ownership, automate checks where possible, and maintain an up-to-date inventory. By following these practices, you ensure your services remain secure and available.
Start by running a verification on one of your current certificates. Record the output, compare it with the expected values, and address any discrepancies. Then, implement monitoring and a checklist to keep TLS health in check. Your future self will thank you.