01 — The 340ms problem
By the summer of 2024, the firm's flagship strategy was quietly bleeding. Their Java matching engine had been the pride of the desk in 2017, but seven years of latency wars had turned it into a liability. During volatility spikes it would balloon to 340ms at the p99 tail — an eternity in a market where the top three competitors were now measuring in single-digit microseconds. The head of engineering flew to our office with a single slide: a chart of arbitrage windows they had missed in Q2, annotated with the dollar figure they'd forfeited to faster rivals. It ended in eight digits.
02 — Rewriting the hot path
We embedded six engineers inside their Chicago floor for the first two weeks — no code, just measurement. We instrumented every stage of the order lifecycle, from NIC interrupt to fill confirmation, and built a flame graph the desk had never seen. The verdict was uncomfortable: GC pauses, cache-line contention, and a scheduler that had no idea NUMA existed. We proposed a full rewrite of the core in Rust, with lock-free ring buffers, NUMA-pinned threads, and market data ingress offloaded to Xilinx FPGAs so risk checks never touched the CPU on the hot path.
03 — Shadow trading in production
Rewriting an engine that touches real money is a special kind of terrifying. So before a single order routed through the new stack, we built a shadow harness that replayed 90 days of live prod traffic through the Rust engine in parallel with the Java one. Every divergence — every fill, every rejection, every microsecond — was logged and reviewed. It caught fourteen bugs the unit tests never would have. By the time we flipped the first venue live, the desk had watched the new engine trade shadow-P&L to within 3 basis points of the real book for six weeks straight.
04 — The eleven-day cutover
We rolled venue by venue behind a feature-flagged router. The riskiest one was scheduled for a Tuesday morning. The desk lead sat behind us with his hand hovering over the kill switch and did not once press it. Eleven days later the Java engine served its final order and was decommissioned. The new stack held 4.2M events per second at peak and delivered 8ms at the p99 — a forty-times improvement — on 38% less hardware.