Back to All Scenarios
PASSEDcloud / helm_chart_rollback_failure
Helm Chart Rollback Failure — Stuck Between Versions
A Helm chart upgrade fails partway through due to a resource conflict, and the automatic rollback also fails because the previous release's CRDs are incompatible with the current cluster state. The release is stuck in a 'pending-rollback' state. Kubernetes resources are in a mixed state between the old and new versions.
Pattern
CONTAINER_EVENT
Severity
CRITICAL
Confidence
95%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | CONTAINER_EVENT | CONTAINER_EVENT | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 29 linked | |
| Cascade Escalation | N/A | No | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
Helm 3.14. Release 'api-gateway' in namespace 'production'. Upgrade from v2.3 to v3.0 (breaking CRD changes). Upgrade failed at 60% (3 of 5 resource types applied). Rollback failed: CRD version conflict. Release status: pending-rollback. Mixed resource state.
Injected Error Messages (2)
Helm release 'api-gateway' stuck in pending-rollback — upgrade from v2.3 to v3.0 failed: 'resource conflict: CustomResourceDefinition gateway.networking.k8s.io already exists with incompatible schema', rollback also failed: 'cannot patch CRD: invalid: spec.versions: must have exactly one storage version', release history shows 3 failed revisions, docker images mixed between v2.3 and v3.0 across replicas, kubelet reporting pod restart failures
API gateway routing partially broken — Kong gateway pods running mixed versions (2 pods v2.3, 1 pod v3.0), CRD 'KongPlugin' schema mismatch causing plugin configuration errors, 30% of API routes returning 'no Route matched', rate limiting plugin disabled due to CRD version conflict, container restart loop on v3.0 pods, external traffic partially disrupted
Neural Engine Root Cause Analysis
The Helm release 'api-gateway' is stuck in pending-rollback state due to a failed upgrade from v2.3 to v3.0 caused by incompatible CustomResourceDefinition (CRD) schema conflicts. The upgrade introduced breaking changes to the gateway.networking.k8s.io CRD that cannot coexist with the existing v2.3 schema, and the automated rollback is also failing because the CRD spec.versions configuration is invalid (lacking exactly one storage version). This has resulted in a mixed deployment state with inconsistent container versions across replicas and cascading pod restart failures, explaining the 14 correlated incidents.
Remediation Plan
1. Immediately stop any ongoing rollback attempts to prevent further state corruption. 2. Manually delete the problematic CRD 'gateway.networking.k8s.io' using kubectl (this will temporarily remove gateway configs). 3. Force delete the stuck Helm release using 'helm delete api-gateway --no-hooks'. 4. Clean up any orphaned pods and resources from the failed deployment. 5. Reinstall the working v2.3 release from scratch to restore service. 6. Plan a proper v3.0 upgrade strategy that includes pre-upgrade CRD migration scripts and compatibility testing in a staging environment.
Tested: 2026-03-30Monitors: 2 | Incidents: 2Test ID: cmncjvxtt0562obqe0y3v4qqt