The primary API is green. Should all users return?
Alder Membership used a limited backup service during an outage. The primary API now responds, and the incident channel wants all traffic moved back.
The question
Fictional exercise. Alder manages event memberships, not bank accounts. Every service state and threshold is authored.
- Backup serviceExisting members can show entry passes. Profile changes are held; members see that limitation.
- Primary checksAPI probes pass in East and West. North has no fresh customer-path sample.
- QueuePending membership changes have a 110-second oldest-item age. Return guardrail: no more than 60 seconds.
- Data stateA backup event log still has unapplied records. Duplicate handling and the primary readback must be verified before full return.
The incident owner may approve a limited East-only return after its membership path and data checks pass. Full return requires fresh customer-path evidence in every region, queue age at or below 60 seconds, and verified event-log reconciliation. Pause the return if new missing or duplicate memberships appear. The API becoming reachable does not trigger automatic return.
“What can you claim now, what would you test next, and who should own the return?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Name the current service scope
What would you do here? Open the move you’d make.
Report limited service and the missing checks
The strong move. This preserves both useful service and the known gaps.
Say every function has recovered
Tempting, but it misses something. Profile changes are held and North lacks a fresh sample.
Switch off the backup when the API responds
Tempting, but it misses something. The policy requires a deliberate return decision.
See how a strong candidate says it
“The backup preserves entry passes but not profile changes. I would state that limited service clearly. Two successful API probes do not establish that all membership functions or regions have recovered.”
Why it works: Service scope must match what customers can actually do.
Step 2: Check data before moving the work
What would you do here? Open the move you’d make.
Treat HTTP success as data reconciliation
Tempting, but it misses something. A successful response does not prove the event log was applied correctly.
Delete the event log to remove the backlog
Tempting, but it misses something. That discards the very records needed to restore customer state.
Verify applied records and membership results
The strong move. This checks the customer effect rather than just the transport.
See how a strong candidate says it
“I would inspect event-log reconciliation, duplicate handling and the customer membership readback. A green API can return inconsistent state while pending records remain unapplied.”
Why it works: Reachability and correct customer state are separate checks.
Step 3: Use the permitted return scope
What would you do here? Open the move you’d make.
Move all traffic now
Tempting, but it misses something. The full-return checks are not met.
Seek the conditional East-only decision
The strong move. The proposed scope and prerequisites match the written policy.
Move North first to obtain a sample
Tempting, but it misses something. An unmeasured region is not the authorized test cohort.
See how a strong candidate says it
“Full return is not ready: queue age is 110 seconds against 60, and North is unmeasured. I would ask the incident owner to approve an East-only return only after East's membership and data checks pass, with a pause on missing or duplicate memberships.”
Why it works: A conditional small return is not permission for global return.
Step 4: Make the handoff actionable
What would you do here? Open the move you’d make.
Handoff the known state and next checks
The strong move. Each gap has an owner and a condition that can be read back.
Close the incident because East is promising
Tempting, but it misses something. A conditional result in one region cannot establish full recovery.
Write root cause confirmed without further work
Tempting, but it misses something. The packet gives recovery state, not a completed cause investigation.
See how a strong candidate says it
“I would name the incident owner for the return, the data owner for reconciliation and the regional owner for North's sample. The record would retain queue age, pending records, checks completed and the next check time. Full recovery remains unconfirmed.”
Why it works: A handoff needs owners and unresolved conditions, not a reassuring summary.
The answer, in short
- Decision
- Keep full return on hold and assign the missing checks.
- Because
- Queue lag, reconciliation and North's evidence do not yet meet the return policy.
- What would change it
- Verified reconciliation, queue age within its guardrail and fresh customer-path samples in every region could support full return.
Common traps
- Using an API probe as proof of correct customer state.
- Losing missing-region and queue conditions during handoff.
Check your answer
- Did you state which functions are available?
- Did you check data effects separately from reachability?
- Did you apply the actual return scope and stop rule?
- Did the handoff name owners for the remaining gaps?
Last reviewed Nov 10, 2026