Back to All Scenarios
PASSEDdatabase / mongodb_replica_set_election
MongoDB Replica Set Election Storm
A network partition between MongoDB replica set members triggers repeated elections. The primary steps down, but no node can achieve majority quorum due to split-brain networking. Applications receive 'not master' errors and writes fail across all connected services.
Pattern
DATABASE_EVENT
Severity
CRITICAL
Confidence
85%
Remediation
Remote Hands
Test Results
| Metric | Expected | Actual | Result |
|---|---|---|---|
| Pattern Recognition | DATABASE_EVENT | DATABASE_EVENT | |
| Severity Assessment | CRITICAL | CRITICAL | |
| Incident Correlation | Yes | 42 linked | |
| Cascade Escalation | Yes | Yes | |
| Remediation | — | Remote Hands — Corax contacts on-site support via call, email, or API |
Scenario Conditions
MongoDB 7.0 replica set with 3 members (1 primary, 2 secondaries). Network partition isolating the primary from both secondaries. Priority-based election configured. Connection pool size: 100 per app server. 5 application servers.
Injected Error Messages (3)
MongoDB replica set election storm — primary mongo-01 stepped down after losing contact with secondaries, replSetGetStatus: 'state: SECONDARY', election triggered but no candidate can reach majority quorum, 7 elections in 3 minutes, all failing with 'could not find member to sync from', database connection pool exhausted on all app servers
MongoDB secondary mongo-02 cannot elect new primary — replSetGetStatus: 'state: SECONDARY, lastHeartbeatMessage: could not contact mongo-01', votes received: 1 of 2 needed, network partition suspected, oplog window shrinking to 2 hours, write operations completely unavailable across replica set
Order service write failures — MongoServerError: not master and slaveOk=false, 100% of write operations failing, database connection pool churning (247 connections created/destroyed in 60 seconds), order processing halted, read preference 'primary' returning errors, retry logic exhausted after 30 seconds
Neural Engine Root Cause Analysis
MongoDB replica set is experiencing a split-brain scenario where the primary node (mongo-01) lost network connectivity to its secondary replicas, causing it to step down. Without a majority quorum available, no node can successfully become primary, resulting in continuous failed elections. The connection pool exhaustion on application servers is a downstream symptom of the database cluster being unavailable, and the 16 correlated incidents indicate a cascading failure affecting dependent services.
Remediation Plan
1. Immediately check network connectivity between all MongoDB replica set members (mongo-01 and secondaries) 2. Verify MongoDB service status on all replica set nodes 3. If network connectivity is restored, manually trigger replica set reconfiguration to establish quorum 4. If secondaries are down, restart MongoDB services on secondary nodes first, then primary 5. Monitor replica set status until stable primary is elected 6. Restart application services to clear exhausted connection pools 7. Verify all dependent services recover once database is available
Tested: 2026-03-30Monitors: 3 | Incidents: 3Test ID: cmncjtbc804imobqeknl6t8vq