Async-First: Trading Latency for Reliability
Why I push payments, notifications and ingestion onto queues — and the failure modes that decision introduces.
A recurring pattern across my systems is moving anything that talks to a flaky third party off the request path and onto a queue.
Why
- A payment provider webhook, an SMS gateway or an on-chain data source will eventually be slow or down. If that call is synchronous, its failure becomes your failure.
- Queues let you absorb bursts, retry with backoff, and guarantee at-least-once delivery.
In a campervan-rental booking backend, payment processing runs through an async Messenger queue so a slow Adyen call never blocks a booking. In an AI email-automation platform, inbound emails hit a BullMQ pipeline before any LLM ever sees them, so a provider outage smooths into a backlog instead of dropped requests.
The cost you take on
Async isn't free — you trade one class of problem for another:
- Idempotency becomes mandatory. At-least-once delivery means handlers will run twice.
- Ordering is no longer guaranteed; design for it or make operations commutative.
- Observability matters more — a silent backlog is worse than a loud error.
The heuristic: make the operation idempotent first, then make it async. Do it in the other order and you'll debug duplicate charges in production.