Store handoff confusion
Pickers staged bags twice for the same order because couriers arrived with separate dispatch notifications.
Late deliveries
18 customers received delayed ETAs after stores paused handoff to resolve which courier should take the order.
Manual coordination
Support and dispatch teams handled Slack escalation manually for 42 minutes during peak operations.
Timeline
What happened
12:03
Error-rate alert fires for dispatch worker retries.
12:08
Dispatch team reports duplicate courier acceptances in Slack.
12:21
Engineers identify replayed queue acknowledgements after worker restart.
12:45
Replay consumer disabled and affected orders reconciled manually.
Root cause
Primary failure
The courier dispatch worker retried message acknowledgement after a process restart and re-emitted already-sent
dispatch events because deduplication was keyed on attempt_id instead of stable order_dispatch_id.
Detection
How we found it
Automated alerts caught retry spikes, but the customer-facing symptom was surfaced first by dispatch operators in Slack. No dedicated alert existed for duplicate courier acceptances per order.
Remediation
What we did during the incident
| Time | Action | Owner |
|---|---|---|
| 12:14 | Paused replay consumer for courier dispatch queue | On-call engineer |
| 12:18 | Stopped automatic reassignments in dispatch console | Dispatch lead |
| 12:29 | Patched dedupe key to use order dispatch identifier | Backend engineer |
| 12:45 | Reconciled affected orders and resumed queue processing | Ops + engineering |
Customer communication
Outbound notes
Customer message
Apologized for delivery delay, confirmed no duplicate charges, and offered credit for orders delayed beyond 15 minutes.
Courier message
Explained dispatch duplication and reminded couriers to rely on the latest accepted trip in the app before collecting bags.
Stable idempotency key
Use order_dispatch_id across emit, ack, and replay flows instead of attempt-local identifiers.
Due: May 22
Duplicate acceptance alert
Add alert when more than one courier accepts the same order within a five-minute window.
Due: May 24
Dispatch runbook update
Document manual reconciliation steps and store-side wording for duplicate courier arrival.
Due: May 21
Prevention work
Follow-up tasks
Replay consumer contract tests
Add restart + replay integration test that verifies duplicate dispatch events are dropped safely.
Store UI hardening
Display active courier name and latest assignment timestamp on the picker handoff screen to reduce confusion during any future inconsistency.
Incident drill
Run a queue replay failure game day with dispatch and support so the next response path is faster.
Final note
Incident summary
This incident was caused by a correct retry instinct combined with the wrong dedupe identifier. The system behaved safely for money movement, but not for courier assignment. Future prevention depends on stable identifiers, replay-safe workers, and better duplicate detection alerts.