Dr. Aaron Sokasian, CTO at DriveWealth, a global B2B financial technology platform.
Fintech leaders understand the importance of developing high-quality features and products—and successfully rolling them out to users—without causing outages or interruptions. There is nothing worse than unveiling a brand new offering that took tireless months, in some cases years, to develop, only to be met with endless reports of bugs and glitches.
Within an industry like ours, every new integration, deployment and launch is a high-stakes opportunity to showcase our reliability and innovation. Too often, however, I see firms struggling with manual security reviews; getting bogged down in agonizingly slow and overly cautious release cycles; or misjudging the balance needed between speed and safety.
That’s why we’re advocates of canary deployments, a strategy that involves moving slowly to move quickly. It’s a modern approach to software development that enables teams to ship software rapidly while maintaining security and regulatory compliance.
Named after the proverbial canary in the coal mine—a songbird that would warn of danger—canary deployments are a minimal-risk early detection software deployment technique. They involve incremental rollouts to subsets of users—with accompanying testing, polishing, and iterating—ahead of a full version release.
Dev staff can monitor the “song” of the “canary deployment” for potential workflow bottlenecks, problematic feature sets or other problems, correcting issues before they’re pushed to the entire user base.
In anticipation of a new feature, a canary deployment could begin by testing a release with a tiny slice of production traffic, as small as 1% of the user base, while the remainder accesses the existing feature set. This allows for real-world testing with minimal impact.
With observations and evaluations in hand, developers can fine-tune workflows or even roll back immature deployments. As soon as all stakeholders are satisfied with the initial canary deployment, it can be made available in ever-widening (yet controlled) tranches, refined even further and eventually set to encompass the entire user base.
Why Slowing Down Matters In Financial Software
Canary deployments are popular because they provide real-world signals before the “blast radius” of a new software release becomes overly expansive.
In finance, a single defect in a new software release can misprice trades or block settlements. Mistakes can draw the ire of regulators, trigger fines or even result in mandated shutdowns. The reputational damage from buggy software or a half-baked update can be devastating for tech providers, not to mention that every major (or minor) outage means engineers are diverted from building better to fighting fires.
By emphasizing upfront safety, fintech firms can achieve and accelerate long-term velocity.
The compounded effect of fewer rollbacks means more time for new features. An early investment in reliability engineering, with automated tests, gated pipelines and staged environments, pays the dividend of long-term growth and improved client satisfaction.
Upstream Care, Downstream Success
There’s a reason canary deployments are the flagship “slow first” technique, with a variety of highly innovative companies—think Revolut, Robinhood, Netflix and Spotify—all utilizing the strategy.
It’s a technique that reduces downtime, makes rollbacks less painful and produces an enhanced user experience, all while emphasizing deliberate planning to achieve sustainable success.
In fact, it’s a strategy that’s highly suitable for “mission-critical applications,” like banking and healthcare.
Multiple canaries can be released to evaluate different program aspects, while resources can be parceled out more judiciously because they aren’t tied up in preventing (or fixing) full-scale failures.
Building the right workplace culture is another key to turning risk-mitigating canary deployments into robust capabilities and long-term growth.
By emphasizing cautious automation over cowboy fixes, policy-first continuous integration and continuous delivery (CI/CD), versioning and observability-driven development, leading fintech firms can avoid the devastating disruptions and reputational shockwaves that result from a botched release.
Putting Canary To The Test
While a noble strategy to improve processes for apps and other industries, the “moving fast to break things” approach is an ill-fit for the realm of financial services, where there’s no room for recklessness.
In our industry, reliability is paramount. The best fintech firms are innovators, but we’re not going to revolutionize the financial sector by putting our clients in jeopardy.
At DriveWealth, we recently handled a textbook canary deployment. We rolled out an expansion of our capabilities to cover the fixed-income asset class. Since this change touched our entire API stack, the rollout had to be carefully coordinated, creating the perfect opportunity to test the canary deployment. The first version went live for only 2% of traffic, limited to a single cohort of partners. After waiting 24 hours, we monitored the impact with detailed instrumentation and side-by-side production baselines. Fortunately, every metric aligned with expectations, so we doubled exposure to 5% after 12 hours and then continued doubling every few hours until we reached 100%. Within just a couple of days, we successfully delivered this high-impact change to all of our partners with complete confidence. The slower, metered rollout actually let us move faster overall, with compliance sign-off, SRE validation and partner adoption all progressing in parallel instead of in series.
How ‘Slow’ Becomes ‘Fast’
While it might seem like slow and steady could produce a winner, under the right conditions, “slow” can become “fast.”
Canary deployments protect market integrity, support same-day releases, enable hot patches during trading hours (without full systemwide freezes) and keep compliance departments sane by generating vital release logs that can be accessed in the event of an audit.
They provide the ideal balance between control and speed, letting developers test new features in the real world without putting all users at risk. They lend themselves well to failure recovery, in the event of a rollback, and allow for early integration of user feedback.
Perhaps best of all, the higher customer confidence produced by canary deployments accelerates product uptake, while reduced regulatory incidents keep operational risks at a minimum. Fintech software development has never been more seamless.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?