Real-Time Risk Analytics with Apache Beam and Dataflow
Risk analytics in capital markets has traditionally been a batch operation. Run the VaR calculation overnight, get results in the morning, and hope the market does not move during the gap. That model broke down during the 2020 volatility events, when firms discovered that their risk teams were making decisions on data that was hours old. We rebuilt the risk analytics pipeline for a global markets firm using Apache Beam and Google Cloud Dataflow. The result: intraday VaR windows dropped from 3 hours to 14 minutes, and new data feeds were onboarded in 3 weeks instead of 10. Here is how we did it. ...