Kubernetes Migration Challenges: 7 Problems [Solved]

Problem 1: Pod Networking Fails Across Nodes

The first ping after migration fails. Pods on different nodes can't reach each other even though kubectl shows them Running.

This happens when the CNI plugin's pod CIDR overlaps with your existing network ranges or when the plugin isn't installed on every node. I've seen Calico configured with 10.244.0.0/16 collide with an internal office subnet using the same range.

Check the CNI configuration file at /etc/cni/net.d/ on each node. Verify the pod network CIDR doesn't conflict with node IPs, service IPs, or external networks. Run ip route show on a worker node—you should see routes for the pod CIDR pointing to the CNI bridge interface.

If routes are missing, the CNI daemonset probably didn't start. Check with kubectl get pods -n kube-system and look for network plugin pods in CrashLoopBackOff. Common fixes: ensure kernel modules like br_netfilter are loaded (lsmod | grep br_netfilter), and confirm IP forwarding is enabled (sysctl net.ipv4.ip_forward should return 1).

After fixing the CNI config, restart the plugin daemonset. Networking usually recovers within 30 seconds. Test with a simple curl from one pod to another using pod IPs directly before testing service DNS names.

Problem 2: Persistent Volumes Won't Bind

Your stateful workloads sit in Pending because PersistentVolumeClaims never bind to volumes. Event logs say "no persistent volumes available" or "volume node affinity conflict."

Kubernetes requires explicit storage classes with provisioners. If you're coming from a system where storage was just a mount point, this feels like unnecessary ceremony. It's not. The provisioner handles dynamic volume creation and attachment across nodes.

List available storage classes with kubectl get storageclass. If the list is empty, create one that matches your storage backend—an NFS provisioner for shared storage, or a CSI driver for cloud block storage. The access mode matters: ReadWriteOnce works for single-node apps, but databases that might reschedule need ReadWriteMany.

Pre-create PersistentVolumes if you're binding to existing storage. The capacity, access mode, and node affinity must match the claim exactly. I've debugged three-hour delays that came down to a typo in the storage size: the PV offered 100Gi but the PVC requested 100G.

For cloud environments, verify the CSI driver has IAM permissions to create volumes. On AWS, the EBS CSI driver needs ec2:CreateVolume and ec2:AttachVolume. Without those, claims sit unbound forever with no useful error message in the pod events.

• Match PVC access mode to workload needs: RWO for single-pod apps, RWX for shared storage
• Set storage class reclaimPolicy to Retain for production data—Delete will wipe volumes on claim deletion
• Test volume provisioning in a non-production namespace before migrating stateful workloads

Problem 3: RBAC Denials Block Service Accounts

Deployments that worked in staging fail in production with "forbidden: User 'system:serviceaccount:app:default' cannot list resource." The app can't read ConfigMaps, Secrets, or even list its own pods.

Kubernetes defaults to zero trust. The default service account in each namespace has almost no permissions. Applications that expect to query the API server or watch resources need explicit RoleBindings or ClusterRoleBindings.

Start by identifying what your app actually needs. Does it read Secrets? List pods? Watch for config changes? Create the minimum Role that grants those verbs on those resources. Binding cluster-admin to every service account is a security disaster I've had to clean up more than once.

Use kubectl auth can-i to test permissions before deploying: kubectl auth can-i list pods --as=system:serviceaccount:myapp:default. If it returns no, create a Role and RoleBinding. Keep the scope namespace-specific unless the app genuinely needs cluster-wide access.

Audit existing bindings after migration. I found a cluster where 40 service accounts had cluster-admin because someone copied a tutorial without reading it. Run kubectl get clusterrolebindings -o json and look for bindings to powerful roles. Remove anything that doesn't need to be there.

Problem 4: Resource Starvation Crashes Nodes

Nodes become unresponsive under load. Pods evict randomly. The kubelet logs say "node has disk pressure" or "node has memory pressure."

This happens when you migrate workloads without setting resource requests and limits. Kubernetes doesn't know how much CPU or memory a pod needs, so it schedules everything until a node runs out. Then the kernel OOM killer starts shooting processes.

Set requests based on observed usage, not guesses. Run your workload for a week and collect metrics. If a pod averages 200m CPU and 512Mi memory, set requests slightly above that—say 250m and 600Mi. The request reserves capacity; the scheduler won't place the pod on a node that can't provide it.

Set limits to prevent runaway processes from taking down the node. A memory leak in one container shouldn't kill the kubelet. I typically set limits at 1.5x to 2x the request for CPU and exactly at the request for memory (to trigger OOMKill before affecting neighbors).

For disk pressure, ensure logs don't fill the node filesystem. Configure log rotation with maxSize: 10Mi and maxFiles: 3 in the kubelet config. If you're running a logging sidecar, it needs resource limits too—I've seen Fluent Bit consume 2Gi memory when log volume spikes.

• Baseline resource usage in staging for 7 days before setting production requests
• Set memory limits equal to memory requests to fail fast on leaks
• Set CPU limits 1.5-2x CPU requests to allow bursts without throttling normal operation

Problem 5: DNS Lookups Timeout Randomly

Intermittent DNS failures plague 10% of requests. Sometimes service names resolve, sometimes they don't. Logs fill with "no such host" errors that vanish when you test manually.

CoreDNS handles service discovery in Kubernetes, and it can bottleneck under load. Check if CoreDNS pods are CPU-throttled: kubectl top pods -n kube-system. If they're hitting CPU limits during traffic spikes, increase the limits or scale the deployment.

The default CoreDNS configuration caches for 30 seconds. That's fine for stable services but causes stale results when pods churn quickly. Edit the CoreDNS ConfigMap and reduce the TTL to 5 seconds for internal zones if your services redeploy frequently.

Client-side DNS settings matter too. If your pods use ndots:5 in /etc/resolv.conf (the Kubernetes default), every single-label lookup tries five suffixes before succeeding. A lookup for "api" becomes five queries. Lower ndots to 1 or 2 unless you need deep namespace search paths.

For high-traffic services, consider NodeLocal DNSCache. It runs a DNS cache on every node, reducing load on CoreDNS and cutting query latency by 80%. Deploy it as a daemonset—setup takes 10 minutes and the performance gain is immediate.

Problem 6: Monitoring Loses Visibility After Cutover

Your dashboards go dark after migration. Prometheus scrapes fail. Logs stop flowing to your aggregator. You're flying blind in production.

Kubernetes changes how metrics are exposed. Instead of scraping a static host:port, Prometheus needs to discover pods dynamically using service discovery. Update your prometheus.yml to use the kubernetes_sd_configs with role: pod and annotation-based filtering.

Add prometheus.io/scrape: 'true', prometheus.io/port, and prometheus.io/path annotations to your pod specs. Prometheus will find them automatically. Without these annotations, pods are invisible to scraping even if the metrics endpoint works perfectly.

Log forwarding breaks because container logs go to stdout instead of files. Update your log shipper (Fluentd, Fluent Bit, or Filebeat) to read from /var/log/containers/ on each node. This directory contains symlinks to all container logs.

Set up kube-state-metrics if you don't have it. It exposes cluster-level metrics about deployments, pods, and nodes that Prometheus alone can't see. I check pod restart counts, deployment rollout status, and PVC binding state from kube-state-metrics every single day.

• Annotate every pod template with Prometheus scrape settings
• Update log shipper configs to read /var/log/containers/ instead of application log paths
• Deploy kube-state-metrics for cluster health visibility

Problem 7: Rollbacks Take Too Long When Things Break

Something goes wrong at 3 AM. You need to roll back but the process is manual and error-prone. By the time you've reverted configs and restarted services, downtime has stretched to 40 minutes.

Kubernetes makes rollbacks fast if you design for them. Use Deployments with rolling updates instead of manually managing pods. Keep at least two previous ReplicaSets around (revisionHistoryLimit: 3 in the deployment spec).

Practice the rollback before you need it. Run kubectl rollout undo deployment/myapp in staging and time how long it takes. If it's slower than your SLA allows, investigate why—usually it's slow image pulls or aggressive readiness probes that delay pod readiness.

For critical services, run both the old and new versions simultaneously during migration. Use a service mesh like Istio or Linkerd to split traffic 90/10, then 50/50, then fully cut over. If the new version fails, you just shift traffic back—no redeployment needed.

Document the rollback procedure in a runbook with exact commands. I keep a script called rollback-to-stable.sh in the repo that reverts to the last known-good image tag and scales up the old deployment. When the site is down, you don't want to be searching Slack history for the right kubectl command.

Performance Tuning After Solving Core Issues

Once the blockers are fixed, you can optimize. Start with horizontal pod autoscaling based on CPU or custom metrics. Set the target utilization to 70%—high enough to use resources efficiently, low enough to absorb traffic spikes without queuing.

Tune readiness and liveness probes. A readiness probe that's too fast marks pods ready before they're actually serving traffic. Too slow and you add latency to every deployment. I use 5-second intervals with a 3-second timeout for most HTTP services.

Enable the vertical pod autoscaler in recommendation mode first. It'll suggest better resource requests based on actual usage. After a week, review the suggestions and apply them manually. Full auto-scaling of requests can be risky until you trust the recommendations.

For CPU-bound workloads, check if you're getting throttled. Run kubectl top pods and compare CPU usage to limits. If usage consistently hits the limit, you're throttling. Increase the CPU limit or investigate why the app is consuming more than expected.

Monitor the control plane under load. If API server latency spikes during deployments, you may need to scale the control plane components. On managed Kubernetes services, this is usually automatic. On self-hosted clusters, you'll need to monitor etcd latency and API server request duration, then add replicas when p99 latency exceeds 500ms.

• Set HPA target utilization to 70% for headroom during traffic spikes
• Review VPA recommendations weekly and adjust resource requests manually
• Monitor API server p99 latency; scale control plane if it exceeds 500ms
• Use PodDisruptionBudgets to prevent more than one pod eviction during node maintenance