Checkout errors rose after a release. What do you do first?
Northstar Tickets has a new checkout release on a small cohort. The deployment completed, but customers report failed orders.
The question
Fictional exercise. You are the on-call engineer. All counts, service states and branch results are authored teaching data.
| Failed eligible requests | Eligible requests | |
|---|---|---|
| New r18 / c9 cohort | 380 | 10000 |
| Stable r17 / c8 sample | 9 | 10000 |
Both samples cover 09:00–09:02. The new cohort has 10% of traffic. A green summary last sampled at 08:59.
| Failed eligible requests | Eligible requests | |
|---|---|---|
| r18 / c8 | 8 | 10000 |
| r17 / c9 | 360 | 10000 |
| r18 / c9 | 380 | 10000 |
Same eligible-request definition; these are reproduction samples, not the production cohort totals.
You may pause the rollout and checkout config publisher, restore verified c8 on the affected cohort, and request incident-command support. Binary changes, database writes and global restarts are not authorized. After a scoped config restore, stop and escalate if errors exceed 1% in the next two-minute check.
- Schema
- r17 and r18 both read schema s12. The only schema change added an optional field.
- Restore
- c8 passed a restore rehearsal.
- Publisher
- c9 is republished every minute.
- Config readbackFor the scoped restore only: publisher paused; applied hash matches c8.
- 09:10 customer sample7 failures in 10,000 eligible requests over two minutes. Queue age is 110 seconds; its guardrail is 60 seconds. One region has no fresh sample.
“What evidence would you use, which action is permitted, and what can you say about recovery?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Establish current impact
What would you do here? Open the move you’d make.
Continue because the dashboard is green
Tempting, but it misses something. Its sample ended before the release and customer impact.
Pause and use fresh cohort evidence
The strong move. The new cohort has current customer failures; pausing limits further exposure.
Restart every region
Tempting, but it misses something. The packet does not authorize a global restart or establish that it would help.
See how a strong candidate says it
“The fresh samples show 3.8% failures in the new cohort and 0.09% in the stable sample. The green summary is stale. I would pause the rollout and open an incident, without claiming the release is the root cause.”
Why it works: The observation window matters as much as the dashboard color.
Step 2: Separate the changed inputs
What would you do here? Open the move you’d make.
Compare release and config separately
The strong move. The combinations distinguish a binary change from a config-dependent symptom.
Call r18 the confirmed root cause
Tempting, but it misses something. r18/c8 is healthy in the provided reproduction.
Ignore configuration because the deploy passed
Tempting, but it misses something. A completed deployment is not a service-health result.
See how a strong candidate says it
“The reproduction has high errors with c9 under both binaries, while r18/c8 has low errors. That makes configuration the stronger lead. It does not prove the complete cause or justify a database change.”
Why it works: A comparison can narrow a hypothesis without proving causation.
Step 3: Choose an authorized restoration
What would you do here? Open the move you’d make.
Roll back the image now
Tempting, but it misses something. Binary mutation is not authorized. Record the proposal as not executed; do not invent its readback.
Restart the database
Tempting, but it misses something. No evidence or authority supports this wider mutation.
Stop the publisher and restore c8
The strong move. This is the permitted, rehearsed action; stopping the publisher prevents c9 from overwriting it.
See how a strong candidate says it
“I would pause the rollout and checkout publisher, then restore verified c8 only on the affected cohort. I would check the applied hash and the next two-minute error sample. A binary rollback is outside this permission and leaves c9 as an unresolved input.”
Why it works: The provided manifest and rehearsal support this bounded config restore.
Step 4: State the remaining recovery work
What would you do here? Open the move you’d make.
Declare all regions recovered
Tempting, but it misses something. One region has no fresh sample and the queue remains above its guardrail.
Report partial recovery and assign the gaps
The strong move. The claim matches the measured scope and leaves a named owner for missing checks.
Resume feature rollout immediately
Tempting, but it misses something. The recovery conditions have not been met.
See how a strong candidate says it
“The scoped restore has a matching hash and 0.07% errors in the observed sample. Queue age still exceeds its guardrail, and one region is unmeasured. I would keep the incident open and assign both checks to the incident owner. Root cause remains under investigation.”
Why it works: Mitigation evidence is different from full recovery.
The answer, in short
- Decision
- Report partial recovery; keep the incident open.
- Because
- The scoped action has readback, but queue lag and a missing region remain.
- What would change it
- Fresh region evidence and queue age within its guardrail could support a later recovery decision.
Common traps
- Using a stale green summary as recovery evidence.
- Inventing readback for a binary rollback that was not authorized.
Check your answer
- Did you use current customer samples?
- Did you separate release and configuration evidence?
- Did you keep the action within written authority?
- Did you retain the queue and region gaps?
Last reviewed Nov 10, 2026