Back to All Scenarios
PASSEDsecurity / jwt_signing_key_rotation_failure
JWT Token Signing Key Rotation Failure — Authentication Broken
A JWT signing key rotation was performed on the authentication service, but the new public key was not distributed to 4 of 7 microservices that validate tokens. These services are rejecting all tokens signed with the new key, while the auth service has stopped issuing tokens with the old key.
Pattern
UNKNOWN
Severity
CRITICAL
Confidence
95%
Remediation
Auto-Heal
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | UNKNOWN | UNKNOWN | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 36 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Auto-Heal — Corax resolves autonomously |
Scenario Conditions
7 microservices validating JWTs. Auth service rotated to new RS256 key. New public key pushed to 3 services only. 4 services still have old public key. All new tokens rejected by 4 services. Old tokens expiring (1-hour lifetime).
Injected Error Messages (3)
JWT signing key rotation partial failure — auth service successfully rotated to new RS256 signing key (kid: 'key-2026-03-29'), JWKS endpoint updated, but key distribution webhook failed for 4 of 7 downstream services, auth service issuing tokens with new key only, old key removed from signing rotation, all tokens issued in last 30 minutes use new key that 4 services cannot verify
user service rejecting all authentication tokens — JWT signature verification failing with 'invalid signature: key ID key-2026-03-29 not found in local keystore', service has old public key (kid: 'key-2025-12-01') only, JWKS refresh mechanism not configured (static key file), 100% of authenticated requests returning 'unauthorized', users unable to access profile, settings, or account management, econnrefused from auth service key distribution endpoint during rotation
order service authentication broken — all API requests returning 'unauthorized: token signature verification failed', JWT verification using stale public key that does not match current signing key, order placement, order history, and order tracking all non-functional for authenticated users, econnrefused from auth service key distribution endpoint during rotation, 4 services affected by incomplete key rotation
Neural Engine Root Cause Analysis
The auth service successfully rotated its JWT signing key but failed to distribute the new public key to 4 out of 7 downstream services via webhook notifications. This created a validation mismatch where the auth service is issuing tokens with a new signing key that these services cannot verify because they lack the updated public key. The old signing key was prematurely removed from rotation, breaking authentication for any service that didn't receive the key update.
Remediation Plan
1. Immediately re-add the previous signing key to the auth service's active key rotation to restore token validation for affected services. 2. Retry webhook distribution for the 4 failed services to deliver the new public key (kid: 'key-2026-03-29'). 3. Verify JWKS endpoint accessibility and validate that all downstream services can fetch updated keys. 4. Once all services confirm receipt of new key, safely remove old key from rotation. 5. Implement webhook retry mechanism and validation checks before key removal in future rotations.
Tested: 2026-03-30Monitors: 3 | Incidents: 3Test ID: cmncki07509phobqe1lb4xv75