Teaching a distributed system to say no, politely
Bloomberg's portfolio analytics platform sits behind clients whose workloads arrive in unpredictable surges. Rather than buy our way out with hardware, I architected CFC, a credit-based flow control system that measures resource usage across an n-tier distributed system and uses historical statistics to regulate flows before they become outages.
CFC cut risk load by roughly 90% and held the platform steady through the onboarding of a new client consuming an additional 30% of bandwidth. The smooth rollout was the result of thorough pre-deployment analysis, which earned support from senior stakeholders. The platform also surfaced traffic patterns and client-usage data that now inform our SRE team, product managers, and the CFO office's commercial model.