Kubernetes Node Autoscaling: 5 Fixes That Work
Fix kubernetes node autoscaling failures by comparing cluster autoscaler vs Karpenter. Stop stuck nodes with these 5 practical configuration checks.

On this page
- Cluster Autoscaler vs. Karpenter: Which One Fits Your Stack?
- Fix 1: Define Explicit Resource Requests on Every Pod
- Fix 2: Align Node Group Min/Max Sizes with Your Workload
- Fix 3: Check Cloud Provider IAM Permissions
- Fix 4: Review Autoscaler Logs for Pending Pods
- Fix 5: Test Autoscaling with a High-Resource Pod
TL;DR — Key takeaways
- Cluster Autoscaler is best for simple, predictable workloads with static node groups, while Karpenter handles complex bin-packing and dynamic provisioning faster.
- The most common cause of stuck autoscaling is a mismatch between pod resource requests and node group capacity limits.
- Always define explicit min/max sizes in your node group configuration to prevent unexpected cost spikes or capacity starvation.
- Verify autoscaling health by checking the 'unhealthy' status of nodes and reviewing the autoscaler's event logs for pending pods.
Kubernetes node autoscaling fails when the gap between what your pods need and what your node groups can provide gets too wide. In support tickets I handled, the usual culprit was a full disk or a misconfigured resource request, not the autoscaler itself.
This guide compares the two main approaches—Cluster Autoscaler and Karpenter—and gives you five concrete fixes to stop stuck nodes and ensure your cluster scales reliably.
Cluster Autoscaler vs. Karpenter: Which One Fits Your Stack?
Cluster Autoscaler (CA) is the older, more established option. It works by watching for pending pods and scaling existing node groups up or down. It’s simple, well-documented, and integrates tightly with most cloud providers. However, it’s limited to the node groups you define upfront. If your workload needs a specific instance type that isn’t in your node group, CA can’t help.
Karpenter is newer and more dynamic. It doesn’t rely on pre-defined node groups. Instead, it looks at pending pods, calculates the best instance type to satisfy their resource requests, and provisions a node directly. This allows for better bin-packing and lower costs, especially for variable workloads. But it’s more complex to configure and debug.
So what if you’re not sure which one to pick? If your workloads are predictable and you want a simple setup, stick with CA. If you’re running diverse, spiky workloads and want to optimize costs, Karpenter is worth the extra effort.
- CA: Best for static, predictable workloads.
- Karpenter: Best for dynamic, cost-optimized workloads.
- CA: Easier to debug with standard node group logs.
- Karpenter: Requires deeper understanding of instance types and pricing.
Fix 1: Define Explicit Resource Requests on Every Pod
This is the number one cause of autoscaling failures. If a pod doesn’t have a CPU or memory request, the autoscaler can’t calculate how much capacity it needs. It might assume the pod needs zero resources, leading to under-provisioning and eventual node saturation.
Check your pod specs. Every container should have a `resources.requests` block. If you’re using Helm charts or Kustomize, ensure your default values include these requests. A quick `kubectl get pods -o json | jq '.items[].spec.containers[].resources.requests'` will show you which pods are missing them.
In support tickets I handled, the usual culprit was a full disk or a misconfigured resource request, not the autoscaler itself. Fix the requests first, then test the autoscaler.
- Every container must have CPU and memory requests.
- Use `kubectl get pods -o json` to audit missing requests.
- Set requests to at least 50% of your expected peak usage.
Fix 2: Align Node Group Min/Max Sizes with Your Workload
If your node group’s max size is too low, the autoscaler will hit a ceiling and stop scaling up. If the min size is too high, you’re paying for idle capacity. The key is to match these limits to your actual traffic patterns.
Start with a min size that covers your baseline load. Then set the max size to handle your peak load plus a 20% buffer. For example, if your baseline is 5 nodes and your peak is 20, set min=5 and max=24. This gives you room to breathe without over-provisioning.
Monitor your node group utilization over a week. If you’re consistently hitting the max, increase it. If you’re rarely using more than the min, decrease it. This iterative tuning is more effective than guessing.
- Min size: Cover your baseline load.
- Max size: Peak load + 20% buffer.
- Monitor utilization for at least one week before adjusting.
Fix 3: Check Cloud Provider IAM Permissions
The autoscaler needs permission to create and delete nodes. If the IAM role attached to your cluster lacks these permissions, the autoscaler will fail silently or throw cryptic errors. This is a common issue when migrating clusters or using new cloud accounts.
For AWS, ensure the role has `ec2:RunInstances`, `ec2:TerminateInstances`, and `ec2:DescribeInstances` permissions. For GCP, check the `roles/compute.instanceAdmin` and `roles/compute.nodeAdmin` roles. For Azure, verify the `Virtual Machine Contributor` role.
Test the permissions by manually creating a node in the same subnet as your node group. If that fails, the autoscaler will too. Fix the IAM role, then retry the autoscaling operation.
- AWS: `ec2:RunInstances`, `ec2:TerminateInstances`, `ec2:DescribeInstances`.
- GCP: `roles/compute.instanceAdmin`, `roles/compute.nodeAdmin`.
- Azure: `Virtual Machine Contributor` role.
- Test by manually creating a node in the same subnet.
Fix 4: Review Autoscaler Logs for Pending Pods
The autoscaler’s logs are your best friend when things go wrong. Look for messages like `FailedScheduling`, `Insufficient resources`, or `Unable to add node`. These indicate that the autoscaler tried to scale up but couldn’t find a suitable node group or instance type.
For Cluster Autoscaler, check the logs of the `cluster-autoscaler` pod. For Karpenter, check the `karpenter` controller logs. Use `kubectl logs -n kube-system <pod-name>` to view them. Filter for errors related to your specific node group or instance type.
If you see `Insufficient resources`, it means the autoscaler couldn’t find a node group with enough capacity. Check your node group configuration and resource requests. If you see `Unable to add node`, it’s likely an IAM or network issue.
- Check `cluster-autoscaler` or `karpenter` pod logs.
- Look for `FailedScheduling` and `Insufficient resources` errors.
- Filter logs by node group or instance type for faster diagnosis.
Fix 5: Test Autoscaling with a High-Resource Pod
Don’t wait for a production incident to test your autoscaler. Deploy a test pod with high CPU and memory requests to trigger a scale-up. Use a pod that requests more resources than your current node group can provide. This forces the autoscaler to create a new node.
For example, if your node group has 4 vCPUs and 16GB RAM, deploy a pod requesting 8 vCPUs and 32GB RAM. The autoscaler should detect the pending pod and scale up. Monitor the node group’s size and the pod’s status. If the pod becomes `Running`, the autoscaler is working.
Remember to clean up the test pod and node afterward. Use `kubectl delete pod <pod-name>` and let the autoscaler scale down the node. This ensures you’re not paying for idle capacity.
- Deploy a test pod with high resource requests.
- Monitor the node group’s size and the pod’s status.
- Clean up the test pod and node afterward to avoid cost.
Quick troubleshooting checklist
- Verify that all pods have explicit CPU and memory requests defined.
- Check that node group min/max sizes align with your expected workload variance.
- Review the autoscaler logs for 'FailedScheduling' or 'Insufficient resources' errors.
- Ensure the cloud provider's IAM roles have permission to create and delete nodes.
- Test the autoscaler by deploying a pod with high resource requests to trigger scale-up.
FAQ
Why is my Kubernetes cluster not scaling up nodes?
Usually, pods lack explicit resource requests, or the node group has reached its maximum size limit. Check the autoscaler logs for 'Insufficient resources' and verify that the cloud provider's API permissions allow node creation.
What is the difference between Cluster Autoscaler and Karpenter?
Cluster Autoscaler works with pre-defined node groups and scales them up or down. Karpenter provisions nodes directly based on pod requirements, allowing for more flexible and faster bin-packing of workloads onto the most cost-effective instance types.
How do I prevent my Kubernetes cluster from scaling down too aggressively?
Set a higher minimum size for your node groups and configure the autoscaler's 'scale-down-unneeded-time' parameter to a longer duration. This ensures nodes remain available for short-lived workloads or sudden traffic spikes.
Related articles
- Hosting OperationsKubernetes Cluster Autoscaling: 3 Configs That Cut CostsConfigure HPA, VPA, and Cluster Autoscaler to match real traffic patterns and reduce cloud spend by up to 40% without performance loss.
- Hosting OperationsEmail Not Sending? 7 Fixes That Actually WorkFix email not sending issues fast. Check MX records, SPF, and logs. This guide covers 7 practical steps to restore outbound mail flow.
- Hosting OperationsWebsite Down? 7 Security Fixes That Actually WorkStop recurring website down errors by hardening your server. Fix DNS, SSL, and firewall misconfigurations with these 7 verified steps.
- Hosting OperationsWebsite Down? 7 Security Fixes That WorkFix a website down by checking DNS, disk space, and firewall rules. Secure your server with these 7 hardening steps to prevent future outages.
- Hosting OperationsEmail Not Sending: 7 Fixes That WorkStop email not sending issues fast. Compare SMTP, relay, and local delivery, then follow clear steps to restore mail flow.
- Hosting OperationsKubernetes Cluster Troubleshooting: 8 CrashLoop FixesFix CrashLoopBackOff, ImagePullBackOff, and OOMKilled errors in your Kubernetes cluster. Eight proven steps with kubectl commands and security checks.