Why did the deployment finish while jobs stopped?
Ember Build completed a worker rotation. The deployment is green. Customer jobs are waiting to start.
The question
Fictional exercise. You own triage, not global routing authority. All states and notes are authored.
- Routing state35% of new jobs still target group South. South has no available workers; its last worker was removed at 10:02.
- Other groupsNorth and West handle the remaining traffic. They have only 5% spare capacity, not enough to absorb South's full share.
- Alert timelineQueue-wait alert fired at 10:03. It was acknowledged as expected during rotation. Customer queue age has continued to rise.
- Change logWorker removal completed. The earlier route-drain task is still marked pending.
Pause the rotation now. The incident owner can approve restoring South's last tested worker pool. A global route shift requires a separate capacity check and approval. The replacement tool must refuse worker removal while active routing still targets that pool.
“Which explanation fits the states, what should happen now, and which control would prevent a repeat?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Read the customer symptom
What would you do here? Open the move you’d make.
Treat the green deploy as healthy service
Tempting, but it misses something. The deployment reports task completion, not customer job starts.
Suppress the alert for the full maintenance period
Tempting, but it misses something. The observed queue growth is not established as an acceptable effect.
Pause and inspect routing with job starts
The strong move. This joins current user impact to actual routing and worker state.
See how a strong candidate says it
“A completed worker-removal job says nothing about whether queued customer jobs can start. I would compare queue wait, start rate and routing targets at the same time, and pause the rotation.”
Why it works: The deployment result and service result measure different things.
Step 2: Check the proposed explanation
What would you do here? Open the move you’d make.
Compare routing targets with available pools
The strong move. It exposes a target that has no capacity.
Add workers to North only
Tempting, but it misses something. The jobs still target South; more North capacity alone does not fix that state.
Assume the alert is expected because work was planned
Tempting, but it misses something. The pending drain and growing queue contradict that assumption.
See how a strong candidate says it
“South has no workers but still receives jobs. Its route-drain task is pending. This supports an ordering failure. I would not call it insufficient global capacity without comparing target membership and available workers.”
Why it works: The packet contains a direct state mismatch.
Step 3: Restore inside the incident boundary
What would you do here? Open the move you’d make.
Move all South traffic to North now
Tempting, but it misses something. The stated spare capacity cannot absorb that share, and approval is missing.
Request the approved South pool restore
The strong move. This is the named tested recovery path with the correct owner.
Delete waiting jobs to clear the chart
Tempting, but it misses something. Deleting customer work would hide impact rather than restore service.
See how a strong candidate says it
“I would ask the incident owner to approve the last tested South pool, then read back both membership and routing. Sending all South traffic to groups with only 5% spare capacity would create a different risk.”
Why it works: A proposed mitigation must fit both authority and available capacity.
Step 4: Prevent the invalid state
What would you do here? Open the move you’d make.
Add a reminder to the rotation checklist only
Tempting, but it misses something. The tool could still complete the same invalid sequence.
Alert only on CPU
Tempting, but it misses something. CPU does not establish that a routing group can start customer work.
Enforce drain before removal
The strong move. The guard prevents active traffic from being sent to an empty pool.
See how a strong candidate says it
“Worker removal must require confirmed route drain at the actual state owner. A failure test should attempt removal while South remains an active target and verify that removal is refused. A success test should drain first, then verify removal and customer starts.”
Why it works: An enforced ordering rule is stronger than a reminder.
The answer, in short
- Decision
- Pause the rotation and restore a tested South pool with the incident owner.
- Because
- The routing target and available worker state disagree.
- What would change it
- A verified alternative route with enough spare capacity and explicit approval could change the mitigation.
Common traps
- Dismissing a customer alert because maintenance was planned.
- Adding a checklist without preventing the invalid state.
Check your answer
- Did you distinguish task completion from customer service?
- Did you inspect both targets and pool membership?
- Did you check capacity before rerouting?
- Did you propose a failure test for the removal boundary?
Last reviewed Nov 10, 2026