A global markets firm needed faster visibility into credit and market exposures. Risk teams were stuck waiting hours for batch results, quants feared ripping out their C++ models, and technology leaders knew the next market shock would expose the cracks. cloudlogic.dev partnered with the quant and data engineering teams to rebuild the risk pipeline on Apache Beam and Google Cloud Dataflow, enabling near-real-time analytics while keeping legacy quant libraries in play.
Where we started
The existing risk stack was a tangle of hourly batch jobs, brittle C++ quant libraries, and manual data wiring. It left the organisation reacting to market swings rather than anticipating them.
- Traders and CROs only saw exposures once an hour, so fast markets meant flying blind.
- Every attempt to touch the C++ libraries triggered crashes or weeks of regression testing.
- New data sources required hand-rolled plumbing, multiplying code paths and model drift.
- Modernisation efforts kept stalling because no one wanted to sacrifice trusted pricing logic.
The business needed intraday visibility and continuous delivery of analytics without discarding years of quant investment.
What we did
- Introduced a unified Beam pipeline: Delivered a portable compute model that runs the same pipelines on Dataflow and on-prem runners for fallback scenarios. Risk analysts could run historical, intraday, and streaming jobs from a single codebase.
- Wrapped quant libraries safely: Built a JNI-based isolation pattern with protocol buffers, dead-letter handling, and health checks to protect the Beam workers from native crashes. Quants kept their pricing models, engineers gained stability.
- Automated operations: Implemented Terraform and GitHub Actions for infrastructure and pipelines, plus golden dashboards and SLOs that measure data freshness, end-to-end latency, and job reliability. Incident response moved from guesswork to alerts.
Impact
- Cut VaR and stress-report generation from 3 hours to 14 minutes, giving the CRO real-time situational awareness during market swings.
- Enabled intraday streaming analytics for selected desks without duplicating code, unlocking entirely new use cases like rapid liquidity monitoring.
- Reduced onboarding time for new data feeds from 10 weeks to 3, thanks to standardised connectors and a data mesh-ready schema library.
- Improved model explainability through consistent protobuf schemas and lineage reporting, satisfying model risk, audit, and regulators in one go.
“We didn’t just make the graph shorter—we made risk part of the trading day. The team stopped firefighting and started delivering new analytics within sprints.”
— Head of Quant Engineering
How the firm works today
Risk, finance, and technology teams now consume the same data products, with Dataflow handling elasticity in the cloud. Standardised automation and observability keep compute spend under control while satisfying model governance, letting the firm respond to market stress with confidence. The unified pipeline has become the backbone for new AI-driven risk models and forthcoming volatility experiments.