Event-Driven Architectures for Payment Systems

Payment systems are the hardest type of event-driven system to get right. A misplaced message in a recommendation engine means a user sees the wrong product. A misplaced message in a payment system means money moves to the wrong account — and that is a regulatory incident with real financial consequences.

We have designed and built event-driven payment platforms for fintechs processing hundreds of thousands of transactions daily. Here is the pattern that works.

Who Is This Guide For?

This guide is for software architects and platform engineers designing payment systems for fintech and financial services. If you are moving from a synchronous payment flow to an event-driven architecture, this is for you.

By the End of This, You’ll Know…

• How to implement idempotency keys for payment events
• The saga pattern for multi-step payment workflows
• How to achieve exactly-once semantics without sacrificing throughput
• Why outbox patterns prevent dual-write consistency failures

The Idempotency Key Pattern

The fundamental challenge in event-driven payments is duplicate events. A network timeout, a broker restart, or a consumer crash can cause the same event to be delivered twice. If the second delivery triggers a second payment, you have a problem.

The solution is an idempotency key — a unique identifier for each payment event that consumers use to detect and discard duplicates.

1
2
3
4
5

[Payment Request] → Idempotency Key assigned → Event published
                                               ↓
                    [Consumer] → Check idempotency store
                                   ↓ already processed → ACK, no-op
                                   ↓ new → process + record key

The idempotency store must be:

• Durable: Survive consumer restarts and broker failures
• Consistent: Strongly consistent read-after-write to prevent race conditions
• TTL-expiring: Keys should expire after a reasonable window (typically 24-72 hours)

A pattern we use is the outbox table — the idempotency key is written to a database table in the same transaction as the payment state update:

1
2
3
4
5
6
7

BEGIN;
  INSERT INTO payment (id, amount, status, idempotency_key)
  VALUES (gen_random_uuid(), 100.00, 'PENDING', 'key-abc-123');
  
  INSERT INTO outbox (event_id, event_type, payload, created_at)
  VALUES (gen_random_uuid(), 'payment.created', '{"payment_id": "...", "amount": 100.00}', NOW());
COMMIT;

If the consumer crashes before processing, the outbox message is re-published by a background worker. The consumer checks the idempotency store and skips duplicate events.

The Saga Pattern for Multi-Step Payments

A payment flow is rarely a single step. A typical flow: validate → authorise → capture → settle → notify. Each step may involve different services — the ledger service, the risk service, the notification service — and each can fail independently.

The saga pattern orchestrates multi-step workflows with compensating transactions for rollback:

1
2
3
4
5
6
7
8
9

Payment Saga: Orchestrator
  1. Validate → OK
  2. Authorise → OK
  3. Capture → FAIL
     ↓
  Compensate:
  - Reverse authorisation
  - Notify user of failure
  - Log audit event

We implement sagas using a lightweight orchestrator with persistent state, not a distributed transaction. The orchestrator tracks each step’s state in a database and triggers compensating transactions on failure.

Exactly-Once Semantics

Exactly-once delivery in event-driven systems is a myth at the transport level. What you can achieve is effectively-once processing — ensuring that even if an event is delivered multiple times, it produces the same result.

The combination of idempotency keys (for duplicate detection) and outbox patterns (for reliable publication) gives you effectively-once semantics across the entire pipeline.

What You Can Actually Use Today

FAQ

Can I use Kafka transactions for exactly-once payments? Kafka transactions provide exactly-once semantics between producer and broker, and between broker and consumer — but only when the consumer writes to Kafka as its sink. If your consumer writes to a database, you still need an idempotency layer between the database write and the Kafka commit.

What is the difference between choreographed and orchestrated sagas? Choreographed sagas have services reacting to events without a central coordinator. Orchestrated sagas use a central orchestrator (like Temporal or a custom service) to manage the workflow state. For payment systems, orchestrated sagas provide better observability and simpler compensation logic.

How long should idempotency keys live? Until the payment lifecycle is complete. For most payment systems, 72 hours is sufficient — long enough to cover any retry and recovery window, but short enough to avoid unbounded storage growth.

Further Reading

• Event-driven architecture pattern
• Kafka vs Pub/Sub vs NATS

For a deeper discussion of data platform architecture, see our Financial Data Platforms service.