5.The Doorbell and the Mailbox

The Problem in Real Life

Waiting for One Message vs. A Doorbell, a Mailbox Check and a Queue

Webhooks, Polling, Events, Message Queues and Asynchronous Processing

• Polling — checking the mailbox: the consumer asks the provider again and again: "has payment pay_7Hk2 succeeded yet?" Simple and fully under your control, but it wastes requests when nothing has changed, and there's always a delay of up to one polling interval.
• Webhooks — the doorbell: a webhook is the provider calling your URL when something happens: POST https://api.blueticket.example/webhooks/payments with a JSON event like payment.succeeded. Instant and efficient — but you're now the API provider for that endpoint, and you must follow the caller's rules (fast 2xx), and handle duplicates, delays and missing deliveries.
• Securing webhooks — is it really the payment provider? anyone can send a POST to your URL pretending "payment succeeded!". So providers sign each webhook with a shared secret (an HMAC signature in a header). Your handler checks the signature before believing anything (Act 22).
• Events — "this happened": an event is a message saying that something happened: payment.succeeded, order.created, ticket.scanned. In event-driven systems, one part announces events and any number of other parts react to them — the email service, the analytics service, the venue's report — without the announcer knowing who's listening.
• Message queues — the order rail (Act 14): a queue stores messages safely until a worker takes them, and lets work happen later, at the workers' own pace. If a worker crashes, the message stays and is retried. Messages that keep failing go to a dead-letter queue — a "problem shelf" that a person checks, instead of retrying forever.
• Asynchronous processing — answer now, work later: instead of doing slow work inside a request, the server accepts the job, puts it on a queue, and replies right away (often 202 Accepted). Workers do the slow part in the background. The caller gets a fast answer; the work still happens.
• Idempotent processing — the same event twice is fine: because of retries, webhooks are delivered at least once — sometimes twice. So the handler records each event's ID, and if it has already processed evt_88aa, it does nothing the second time. A database unique constraint on the event ID makes this safe even if two copies arrive at the same moment (Act 13).
• Reconciliation — the mailbox check: a scheduled job compares the two systems' records and fixes any difference. Every 5 minutes: "ask the payment provider for every payment that succeeded in the last 2 hours; for each one without an order here, create it." If a webhook is ever lost, this catches it within minutes.

app.post("/webhooks/payments", async (req, res) => {
  // 1. Is it really from the payment provider?
  if (!payments.verifySignature(req.rawBody, req.headers["x-signature"])) {
    return res.status(400).send("bad signature");
  }
  const event = req.body;

  // 2. Save each event once (unique constraint on event_id)
  const isNew = await db.query(
    "INSERT INTO payment_events (event_id, payload) VALUES ($1, $2) ON CONFLICT DO NOTHING",
    [event.id, event]
  ).then((r) => r.rowCount === 1);

  // 3. Queue the slow work, reply immediately
  if (isNew) await queue.publish("payments", { eventId: event.id });
  res.status(200).send("ok");
});

const paid = await payments.listPayments({ status: "succeeded", since: minutesAgo(120) });
for (const payment of paid) {
  const order = await findOrderByPaymentId(payment.id);
  if (!order) {
    await createOrderFromPayment(payment);       // same idempotent code as the worker
    log.warn("Recovered by fallback check", { paymentId: payment.id });
    metrics.increment("payments.recovered");     // alert if > 0
  }
}