Observability in Financial Services: Building Audit-Ready Monitoring at Scale

Observability is not optional in financial services. Regulators require audit trails. Operations teams need to diagnose production issues. Risk teams need to monitor transaction volumes and error rates in real time. A monitoring system that tells you “something is wrong” without telling you what, why, or when is worse than useless — it creates a false sense of security.

We have built observability platforms for banks and fintechs that satisfy regulatory requirements while providing engineering teams with the diagnostic capability they need. The architecture is not about choosing between Datadog, Splunk, or Grafana. It is about designing the data model that makes your observability data useful for both compliance and operations.

Who Is This Guide For?

This guide is for platform engineers, SREs, and compliance leads at financial services firms building or improving their observability stack. If you need observability that satisfies audit requirements while enabling engineering productivity, this is for you.

By the End of This, You’ll Know…

• Why observability in financial services is fundamentally different from observability in tech
• How to design structured logging that satisfies both compliance and engineering requirements
• The distributed tracing patterns that enable end-to-end transaction visibility
• How to build dashboards that serve both operations and compliance stakeholders

The Three Pillars, Reimagined

Traditional observability has three pillars: logs, metrics, and traces. In financial services, these three pillars serve two distinct audiences with different requirements:

Operations team needs:

• Real-time alerting on service degradation
• Diagnostic information for debugging production issues
• Performance metrics for capacity planning

Compliance team needs:

• Complete audit trail for every transaction
• Evidence of control effectiveness
• Historical data for regulatory reporting

The same observability data must serve both audiences. Separate systems for operations and compliance create redundancy and inconsistency.

The cost of poor observability is measurable. A payment processing outage that lasts 30 minutes costs $50,000-$500,000 in direct revenue loss. A compliance violation that results from missing audit logs costs $100,000-$1,000,000 in fines. A security incident that cannot be investigated due to missing logs costs infinitely more in reputational damage. Observability is not a cost centre — it is an insurance policy.

Structured Logging for Compliance

Unstructured logs (free-text messages) are useless for compliance. You cannot query “show me all transactions processed by user X in the last 90 days” in unstructured logs. Structured logging encodes every log entry as a JSON object with defined fields.

Required fields for financial services:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
  "timestamp": "2025-04-15T14:30:00.123Z",
  "level": "INFO",
  "service": "payment-gateway",
  "traceId": "abc-123-def-456",
  "spanId": "span-789",
  "userId": "user-123",
  "accountId": "account-456",
  "transactionId": "txn-789",
  "eventType": "payment.processed",
  "amount": 1500.00,
  "currency": "USD",
  "status": "success",
  "duration": 45,
  "metadata": {
    "correlationId": "req-abc",
    "source": "api-gateway"
  }
}

Key design decisions:

• Trace ID: Every log entry includes a trace ID that links it to the distributed trace. This enables end-to-end transaction visibility.
• Business context: Log entries include business identifiers (user ID, account ID, transaction ID) not just technical identifiers (pod name, container ID).
• Event type: Log entries are categorised by event type, enabling queries like “show me all failed payments in the last 24 hours.”

Distributed Tracing for End-to-End Visibility

Distributed tracing tracks a request as it flows through multiple services. In financial services, a single payment transaction might flow through:

1. API Gateway → Authentication Service
2. Authentication Service → Payment Service
3. Payment Service → Risk Engine
4. Risk Engine → Payment Processor (Stripe/Adyen)
5. Payment Processor → Ledger Service
6. Ledger Service → Notification Service

Without distributed tracing, diagnosing a timeout in this flow requires manually correlating logs across six services. With distributed tracing, you see the entire flow in a single view.

Implementation:

• OpenTelemetry: Vendor-neutral instrumentation. Instrument once, export to any backend.
• Trace context propagation: Pass trace context (trace ID, span ID) in HTTP headers across service boundaries.
• Sampling strategy: For high-volume systems, sample 100% of errors and a percentage of successful requests.

Metrics for Financial Operations

Financial operations metrics are different from typical SRE metrics. They track business outcomes, not just technical health.

Critical metrics:

• Transaction throughput: Number of transactions per second by type (payment, refund, settlement)
• Transaction latency: P50, P95, P99 latency for each transaction type
• Error rate: Percentage of failed transactions by failure type (declined, timeout, system error)
• Reconciliation status: Percentage of transactions successfully reconciled
• Settlement status: Percentage of transactions settled within SLA

Alerting rules:

• Transaction error rate > 1%: Page on-call engineer
• Settlement delay > 2 hours: Page compliance team
• Reconciliation discrepancy > 0.01%: Alert finance team
• Latency P99 > 5 seconds: Page on-call engineer

Dashboard Design

Operations Dashboard

For the SRE team:

• Real-time transaction throughput and error rates
• Service health status (green/yellow/red)
• Latency percentiles by service
• Active alerts and their severity

Compliance Dashboard

For the compliance team:

• Transaction volume by type and time period
• Reconciliation status and discrepancies
• Audit trail for specific transactions
• Regulatory reporting metrics

Executive Dashboard

For leadership:

• Daily transaction volume and success rate
• Revenue impact of failed transactions
• Compliance posture (violations, remediations)
• Platform reliability trends

Retention and Archival

Financial services require long data retention:

• PCI DSS: Retain audit logs for at least 1 year, with 3 months immediately available
• SOX: Retain financial records for 7 years
• GDPR: Retain personal data only as long as necessary for the stated purpose

Tiered storage strategy:

• Hot (0-30 days): Full detail in Elasticsearch or ClickHouse
• Warm (30-365 days): Compressed data in S3 or GCS
• Cold (1-7 years): Archival storage in Glacier or Nearline

What You Can Actually Use Today

• OpenTelemetry: Vendor-neutral instrumentation for logs, metrics, and traces
• ClickHouse: Columnar database for high-volume log storage and analytics
• Grafana: Open-source dashboarding with support for Prometheus, Elasticsearch, and ClickHouse
• Loki: Log aggregation system from Grafana Labs, designed for cost-effective log storage

FAQ

How much does observability cost at scale?

For a fintech processing 100 million transactions per month, expect $20,000-$50,000 per month for a complete observability stack (storage, compute, and tooling). Cost reduction strategies include sampling, compression, and tiered storage.

Can we use open-source tools for compliance?

Yes. OpenTelemetry, ClickHouse, Grafana, and Loki are used in production at regulated financial institutions. The key is proper configuration, retention policies, and access controls — not the tool itself.

How do we handle PII in logs?

PII must be masked or encrypted in logs. Use structured logging with field-level masking (e.g., email: "u***@example.com"). Store PII separately from logs if full PII access is required for investigation.

We help financial services firms design observability platforms that satisfy audit requirements while enabling engineering productivity. If you are building or improving your observability stack, get in touch.