## Intro

Kubernetes Services provide a stable abstraction to expose your applications running as Pods. They give you a consistent IP address, a DNS name, and load balancing across healthy Pods. But when a Service is misconfigured, you can hit frustrating errors: connection refused, no endpoints available, DNS resolution failures, or conflicts with existing ClusterIP addresses. This guide walks through the most common Kubernetes Service error messages, explains why they happen, how to diagnose them with practical commands, and how to fix them safely. You will learn to identify selector mismatches, troubleshoot missing endpoints, resolve ClusterIP conflicts, and debug DNS issues. Each section includes concrete command examples, expected outputs, and rollback steps. By the end, you will have a repeatable process to keep your Services healthy.

## Version and Environment Inventory

Before you start troubleshooting, gather information about your Kubernetes cluster and the affected Service. Knowing the version helps you identify known bugs or behavior changes. Run `kubectl version` to see client and server versions.

```text
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
```

Check the Service details with `kubectl get svc <service-name> -o yaml` and inspect the selector, ports, and type. For example:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: default
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: ClusterIP
```

Also verify that the Pods have matching labels using `kubectl get pods --show-labels`. If the selector does not match any Pod labels, the Service will have no endpoints. This is a common cause of errors.

```text
NAME                     READY   STATUS    RESTARTS   AGE   LABELS
my-app-6d4b7c9f5-abcde   1/1     Running   0          10m   app=my-app,version=v1
my-app-6d4b7c9f5-fghij   1/1     Running   0          10m   app=my-app,version=v1
```

Note that the labels on the Pods include the `app=my-app` key-value pair that the Service selector expects. If your selector uses `app: my-app` but the Pods only have `app: myapp` or no `app` label at all, you will end up with no endpoints.

## Safe Configuration Path

Start with a minimal Service configuration and validate it before making any changes. Use `kubectl apply --dry-run=client` to test syntax without applying anything to the cluster.

```bash
kubectl apply -f service.yaml --dry-run=client
```

Expected output:

```text
service/my-service created (dry run)
```

When modifying an existing Service, save a backup of the current configuration first:

```bash
kubectl get svc my-service -o yaml > service-backup.yaml
```

Apply changes in a scoped manner, for example in a test namespace first. Create a namespace with `kubectl create namespace test`, then apply the Service YAML there using `kubectl apply -f service.yaml -n test` to observe behavior before touching production.

A good practice is to use explicit selectors and always specify `targetPort` clearly. Avoid using `targetPort` as a string name that might not match the container port name. For example, if your container exposes a port named `http` on 8080, your Service should reference `targetPort: http` or `targetPort: 8080`. If you use `targetPort: 8081` but the container listens on 8080, you will get connection refused.

## Verification and Diagnostics

After applying a Service, check its status and endpoints. Run `kubectl describe svc my-service` to see events and endpoints. Example output:

```text
Name:              my-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=my-app
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.0.100
IPs:               10.96.0.100
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         172.17.0.5:8080,172.17.0.6:8080
Session Affinity:  None
Events:            <none>
```

If `Endpoints` is empty, the selector may be wrong. Verify with `kubectl get endpoints my-service`.

```text
NAME         ENDPOINTS                     AGE
my-service   172.17.0.5:8080,172.17.0.6:8080   10m
```

If you see `<none>` under ENDPOINTS, that indicates no Pods are selected. Check the selector and the Pod labels as described above.

Test connectivity from another Pod using `kubectl run -it --rm debug --image=busybox -- sh` and then run `wget -qO- http://my-service`. For DNS, use `nslookup my-service` inside the debug Pod.

```bash
kubectl run -it --rm debug --image=busybox -- sh
# Inside the debug Pod:
wget -qO- http://my-service
nslookup my-service
```

If the Service is in a different namespace than the Pod, use the fully qualified domain name (FQDN) like `my-service.default.svc.cluster.local`. For example, from a Pod in the `test` namespace, you would use `wget -qO- http://my-service.default.svc.cluster.local`.

## Failure Modes and Recovery

### No endpoints available

The most common cause of `no endpoints available` is a selector mismatch between the Service and the Pods. For example, if your Service selector is `app: my-app` but your Pods are labeled `app: myapp` or `tier: frontend`, the Service will not select any Pods and its endpoints will be empty. To fix it, correct the selector in the Service YAML to match the Pod labels exactly.

```yaml
spec:
  selector:
    app: my-app
    # Ensure this matches your Pod labels; e.g., if Pods also have version=v1, you may not need it.
```

After updating, verify with `kubectl get endpoints my-service`. If you need to roll back, reapply the backup configuration with `kubectl apply -f service-backup.yaml`.

Another cause of no endpoints is Pods not being ready. Check Pod status with `kubectl get pods -l app=my-app`. If Pods are in `CrashLoopBackOff` or not ready, the Service will not route traffic to them. Investigate Pod logs with `kubectl logs <pod-name>`.

### ClusterIP conflict

If a Service is created with a specific `clusterIP` that already exists, you get an error like `provided IP is already allocated`. To avoid this, remove the `clusterIP` field from your Service YAML and let Kubernetes allocate a new IP automatically, or delete the conflicting Service.

Error example:

```text
Error from server (AlreadyExists): services "my-service" already exists
```

If you see a more specific error about IP allocation during creation, ensure the `clusterIP` field is not set or choose a different IP within the service CIDR. You can check allocated ClusterIPs with `kubectl get svc --all-namespaces -o wide`.

To resolve:

1. Edit the Service: `kubectl edit svc my-service` and remove the `clusterIP` line, then save. Kubernetes will allocate a new IP.
2. Alternatively, delete and recreate the Service without specifying `clusterIP`:

```bash
kubectl delete svc my-service
kubectl apply -f service.yaml  # service.yaml without clusterIP
```

### DNS resolution failure

If the Service name cannot be resolved, check that CoreDNS is running and that the Service is in the correct namespace. The full DNS name is `<service-name>.<namespace>.svc.cluster.local`. From inside a Pod, run `nslookup my-service`. If it fails, check CoreDNS pods:

```bash
kubectl get pods -n kube-system -l k8s-app=kube-dns
```

Expected output includes CoreDNS pods in Running state. Also verify that the Service exists in the intended namespace: `kubectl get svc -n <namespace>`.

If DNS fails only for external names, check CoreDNS configuration for forwarders. For internal Services, ensure the Service is of type ClusterIP or Headless, not ExternalName (which requires different DNS behavior).

### Connection refused

`Connection refused` typically means the target port is not listening. Ensure the container is listening on the `targetPort` and that network policies allow traffic. Use `kubectl get svc my-service -o yaml` to confirm `targetPort` matches the container port.

For example, if your container listens on port 8080 but the Service `targetPort` is 8081, you will see connection refused. Fix the `targetPort` to 8080. Also check that the container process is actually running and bound to the correct interface (0.0.0.0, not localhost). Use `kubectl exec <pod-name> -- netstat -tulpn` (if available) to see listening ports.

Network policies can also block traffic. If you have a NetworkPolicy that does not allow ingress to the Pods on the target port, you may get connection refused or timeout. Review network policies in the namespace with `kubectl get networkpolicies`.

## Operations Checklist

Use this checklist before and after making Service changes. Replace the placeholders with your specific service name and namespace.

| Task | Command | Expected Result |
|---|---|---|
| Verify cluster version | `kubectl version` | Client and server versions shown |
| Inspect current Service YAML | `kubectl get svc my-service -o yaml` | YAML with selector and ports |
| List Pods with labels | `kubectl get pods -n default --show-labels` | Pods labels match Service selector |
| Dry-run apply | `kubectl apply -f service.yaml --dry-run=client` | No errors: `service/my-service created (dry run)` |
| Describe Service | `kubectl describe svc my-service` | Endpoints section lists Pod IP:port pairs |
| Check endpoints | `kubectl get endpoints my-service` | ENDPOINTS column shows at least one IP:port |
| Test DNS resolution | `kubectl run -it --rm debug --image=busybox -- nslookup my-service` | Output shows ClusterIP for my-service.default.svc.cluster.local |
| Test HTTP connectivity | `kubectl run -it --rm debug --image=busybox -- wget -qO- http://my-service` | Returns expected HTTP response body |
| Check CoreDNS status (if DNS fails) | `kubectl get pods -n kube-system -l k8s-app=kube-dns` | CoreDNS pods are Running and Ready |
| Rollback if needed | `kubectl apply -f service-backup.yaml` | Service returns to previous working state |

Regularly review Services for unused or misconfigured entries. Use `kubectl get svc --all-namespaces` to list all Services and look for those with no endpoints or selector issues. Clean up stale Services to avoid confusion and potential conflicts.

## Conclusion

Kubernetes Service errors can be debugged efficiently by methodically checking the Service definition, selector labels, endpoints, and DNS. Always start with a minimal configuration, use dry-run for validation, and keep backups for rollback. Use the verification commands in this guide to confirm that your Service is working as expected. By following the operations checklist, you can maintain healthy Services and avoid common pitfalls such as selector mismatches, ClusterIP conflicts, and DNS failures. With this repeatable process, you will reduce downtime and keep your applications reliably exposed within your cluster.