Upgrade EKS to Kubernetes 1.29 #12
Conversation
- Update default kubernetes_version in EKS module from 1.28 to 1.29 - Update production EKS cluster to use Kubernetes 1.29 - Aligns with latest EKS supported versions
WalkthroughKubernetes version updated from 1.28 to 1.29 across EKS infrastructure configuration files. The module variable default and production environment configuration both reflect this version bump, while all other settings remain unchanged. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
prod/eks/terragrunt.hcl (1)
19-33: Explicitly specify addon versions for Kubernetes 1.29 compatibility in production.The terragrunt configuration does not specify versions for critical EKS addons (vpc_cni, coredns, kube_proxy), leaving them to default to null. This allows AWS to automatically select versions, but for production clusters, explicitly pinning addon versions to tested and compatible releases is essential to prevent potential networking and DNS failures.
Add the following addon versions to the prod/eks/terragrunt.hcl inputs block:
vpc_cni_version = "v1.20.4-eksbuild.1"coredns_version = "v1.11.4-eksbuild.24"kube_proxy_version = "v1.29.15-eksbuild.16"The module already supports these variables; they simply need to be explicitly provided.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
modules/eks/variables.tf(1 hunks)prod/eks/terragrunt.hcl(1 hunks)
🧰 Additional context used
🔍 Remote MCP Exa, Linear, Notion
Summary of Findings
Based on comprehensive searches across multiple sources, here are the relevant facts for reviewing this Kubernetes 1.29 upgrade PR:
Critical Policy Conflict ⚠️
The EKS Version Management Policy document requires that all production EKS clusters must run Kubernetes version 1.28 until Q2 2025, and specifies that production clusters can only use version 1.28. The PR updates the production cluster to 1.29, which directly conflicts with this documented policy.
Kubernetes 1.29 Technical Details
Amazon EKS announced support for Kubernetes 1.29, which was released with the theme "Mandala". However, important upgrade prerequisites must be addressed:
Breaking Changes:
- The deprecated
flowcontrol.apiserver.k8s.io/v1beta2API version of FlowSchema and PriorityLevelConfiguration are no longer served in Kubernetes v1.29. If any existing manifests or client software use this deprecated API, they must be updated before the upgrade. - New features in 1.29 include the general availability of ReadWriteOncePods access mode for PersistentVolumes, and the LegacyServiceAccountTokenCleanUp feature that labels legacy auto-generated secret-based tokens as invalid.
Required Upgrade Steps
Any production Kubernetes version upgrade requires: Platform Team approval, Security review sign-off, a 2-week staging validation period, Change Advisory Board (CAB) approval, and a scheduled maintenance window.
Linear Search Results
The Linear search returned upgrade-related tickets (subscription/plan upgrades), but no infrastructure or EKS-related issues directly tied to this pull request.
Key Review Recommendations
- Policy Exception Required: This PR bypasses the documented Q2 2025 timeline. Confirm whether this is an approved exception with formal stakeholder sign-off.
- API Migration Check: Verify no applications in the cluster use the deprecated
flowcontrol.apiserver.k8s.io/v1beta2API before upgrading. - Compliance with Upgrade Process: Ensure the PR is accompanied by evidence of Platform Team, Security, and CAB approvals as required by policy.
- Development/Staging Staging: Confirm 1.29 has been validated in development and staging environments before production.
🔇 Additional comments (1)
modules/eks/variables.tf (1)
14-14: The module default change is safe because both dev and prod environments explicitly setkubernetes_versionin theirterragrunt.hclfiles. Dev is pinned to "1.28" and prod is pinned to "1.29", so changing the module default will not trigger unintended upgrades. No risk of automatic environment upgrades.Likely an incorrect or invalid review comment.
| inputs = { | ||
| cluster_name = "prod-eks-cluster" | ||
| kubernetes_version = "1.28" | ||
| kubernetes_version = "1.29" |
There was a problem hiding this comment.
Verify no workloads use deprecated Kubernetes APIs before upgrading.
Based on external tools context: Kubernetes 1.29 includes breaking changes - the deprecated flowcontrol.apiserver.k8s.io/v1beta2 API version of FlowSchema and PriorityLevelConfiguration is no longer served. Any existing manifests or client software using this deprecated API must be updated before the upgrade, or the upgrade will break those workloads.
Run the following script to check for usage of deprecated APIs in the cluster:
#!/bin/bash
# Description: Check for deprecated API usage in Kubernetes manifests and cluster resources
# Note: This requires kubectl access to the prod cluster
echo "=== Checking for deprecated flowcontrol API usage in manifests ==="
# Search for deprecated API in local manifests/helm charts
rg -n "flowcontrol\.apiserver\.k8s\.io/v1beta2" --type=yaml -C3
echo ""
echo "=== Checking cluster for deprecated API usage (requires kubectl) ==="
echo "Run these commands with cluster access:"
echo "kubectl get flowschemas.v1beta2.flowcontrol.apiserver.k8s.io 2>&1"
echo "kubectl get prioritylevelconfigurations.v1beta2.flowcontrol.apiserver.k8s.io 2>&1"Based on learnings from external tools: The deprecated API must be migrated before the upgrade can proceed safely.
🤖 Prompt for AI Agents
In prod/eks/terragrunt.hcl around line 19, the kubernetes_version bump to 1.29
will drop flowcontrol.apiserver.k8s.io/v1beta2 support; run the provided
detection commands against local manifests and the cluster to find any usages,
then update any FlowSchema and PriorityLevelConfiguration manifests and Helm
charts to the v1 API (and corresponding field changes) or remove them if
obsolete, update client libraries or operators that emit v1beta2, apply and
validate the converted resources in a staging cluster, and only then proceed
with the K8s version upgrade.
🧩 Analysis chain
🔍 Additional research query:
Search Linear for EKS Version Management Policy and kubernetes version 1.29 upgrade requirements
💡 Result:
No relevant context found from external knowledge sources.
🔍 Additional research query:
Search Notion for EKS Version Management Policy, Kubernetes upgrade policies, or CAB approval records
💡 Result:
Search Results
I found the following documents in your Notion workspace related to your search:
EKS Version Management Policy
Document Details:
- Owner: Platform Engineering Team
- Last Updated: October 2024
- Status: Active
Key Information:
- Current Approved Version: Kubernetes 1.28 LTS
- Production Requirement: All production EKS clusters must run Kubernetes 1.28 until Q2 2025
Version Matrix:
| Environment | Approved Version | Upgrade Window |
|---|---|---|
| Development | 1.28 - 1.29 | Anytime |
| Staging | 1.28 | After dev validation |
| Production | 1.28 only | Q2 2025 (planned) |
Upgrade Process Requirements:
The policy specifies that any production Kubernetes version upgrade requires:
- Platform Team approval
- Security review sign-off
- 2-week staging validation period
- Change Advisory Board (CAB) approval ✓
- Scheduled maintenance window
Other Related Documents Found:
- EKS Auto-Scaling Policy - Defines auto-scaling boundaries for EKS node groups across environments in coordination with FinOps
- AWS Instance Type Standards - Includes Graviton (ARM64) First Policy (Owner: Cloud Infrastructure Team, Last Updated: November 2024)
🏁 Script executed:
# Explore the repository structure to understand the codebase better
git ls-files | head -20Repository: coderabbitai/rabbits-playground
Length of output: 572
🏁 Script executed:
# Check the actual content of the file to understand context
head -30 prod/eks/terragrunt.hclRepository: coderabbitai/rabbits-playground
Length of output: 875
🏁 Script executed:
# Search for any policy documents or approval requirements in the repo
find . -type f -name "*policy*" -o -name "*approval*" -o -name "*VERSION*" | grep -v ".git"Repository: coderabbitai/rabbits-playground
Length of output: 57
🏁 Script executed:
# Look for any documentation about K8s version requirements
fd -t f -e md -e txt | xargs rg -l "kubernetes.*1\.(28|29)|version.*management" 2>/dev/null | head -10Repository: coderabbitai/rabbits-playground
Length of output: 57
🔍 Additional research query:
Search Linear for policy exceptions, Kubernetes 1.29 upgrade, CAB approval, or prod-eks-cluster related issues
💡 Result:
No relevant context found from external knowledge sources.
🏁 Script executed:
# Check git history to see if there are related commits or PRs
git log --oneline --all -- prod/eks/terragrunt.hcl | head -10Repository: coderabbitai/rabbits-playground
Length of output: 136
🏁 Script executed:
# Look for any approval or exception documentation in the repo
rg -i "exception|approved.*1\.29|kubernetes.*upgrade" --type md --type txt 2>/dev/null | head -20Repository: coderabbitai/rabbits-playground
Length of output: 57
🏁 Script executed:
# Check if there are comments or documentation in the file itself
cat -n prod/eks/terragrunt.hcl | head -40Repository: coderabbitai/rabbits-playground
Length of output: 1135
CRITICAL: This upgrade violates the EKS Version Management Policy.
Your organization's EKS Version Management Policy (Platform Engineering Team, Active) explicitly requires all production EKS clusters to run Kubernetes 1.28 until Q2 2025. This change to 1.29 in prod-eks-cluster directly conflicts with that policy.
The documented upgrade process mandates:
- Platform Team approval
- Security review sign-off
- 2-week staging validation period
- Change Advisory Board (CAB) approval
- Scheduled maintenance window
No evidence of these approvals exists in the PR.
Actions required:
- Obtain formal policy exception with documented stakeholder approval if this upgrade is authorized ahead of Q2 2025
- Provide evidence of required approvals from Platform Team, Security, and CAB
- Confirm staging validation — has 1.29 been running successfully in dev/staging for 2+ weeks?
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.