How Wealthsimple Trade Didn’t Break During GameStop

Yes, there was baking and Netflix. But do you know what else has gone insane in the past year while everyone’s been trapped at home? Trading stocks. So many people, so many stocks, so much exponential growth.

It was a seismic change, but it was focused on one area: this thing called retail investing. Meaning normal people who don’t work in the financial industry and aren’t trading on behalf of hedge funds or mutual funds or ETFs — just regular investors, sitting at home, buying and selling shares of companies they like.

How do we know retail blew up so much? Because we run Canada’s fastest-growing retail investing platform. Wealthsimple Trade, our commission-free stock-trading app, grew by about 1,000% in the past year. And then in just the first six weeks of the year, Trade grew by another 400%. But unlike many platforms, Trade didn’t experience any outages in trading during that time.

Sign up for our weekly non-boring newsletter about money, markets, and more.

Enter your email

Subscribe

By providing your email, you are consenting to receive communications from Wealthsimple Media Inc. Visit our Privacy Policy for more info, or contact us at privacy@wealthsimple.com or 80 Spadina Ave., Toronto, ON.

As people who are interested in the intersection of technology and money, we got excited to dig into exactly why that was and how you re-engineer a company in the midst of a tsunami of new users. So we called up Karney Li, Wealthsimple’s Chief Technology Officer, and Dominique Simoneau-Ritchie, VP of Engineering, and asked them how it all worked.

We knew they were coming. Sort of. As of January, the engineering team at Wealthsimple was expecting growth — because we were already growing. We had a big spring last year, grew more during the early days of lockdown, and the number of Trade users spiked even more dramatically in November and December.

Meanwhile, the engineering team was also looking ahead to February 7, when an ad for Wealthsimple Tax would run during the Super Bowl. They knew from experience, there would be a surge in demand for our services across the board. “We started planning for 10X growth,” explains Li, adding, “Because when you increase by an order of magnitude, that’s when things really start to break.”

In engineering terms, that meant accelerating a transition to “real-time event-driven architecture,” says Simoneau-Ritchie. “This means our systems are able to complete many client requests at the same time, rather than in sequence. And that, in turn, allows us to respond to our customers much more quickly — usually instantly.” In November, for example, the team made a change (for actual engineers out there: they moved a key microservice to an event stream instead of polling), and that led to an 87% increase in how fast Wealthsimple can execute trades when the market opens.

Engineers from across our teams also worked together to brainstorm creative ways to reduce the number of requests across our microservices. The engineers noticed something interesting: many people tend to hold, and watch, many of the same securities. Noticing this pattern, we installed something called Varnish, which is an HTTP reverse-proxy, a web-accelerator (it’s ok if you don’t know what those are, we’ll explain). In engineering terms, Varnish intercepts requests — and, importantly, it can coalesce similar requests, which saves a trip to the server. That strategy allowed the team to reduce the number of actual requests to the server by over 99%. Varnish is also really, really fast. Say a lot of clients are requesting quotes for Air Canada, all at the same time — now the system can fetch that data once and service all the requests with much less delay.

Then the meme-stock phenomenon happened. In mid-January, the GameStop phenomenon — aka the meme-stock bubble — began to sweep the markets. Overnight, Trade was experiencing growth that made 2020 look slow by comparison. In the weeks after New Year’s Day, the app’s user base quadrupled, and we faced a new problem: how do we handle all these new customers?

“In the first week of January, we knew we could handle 9,000 clients concurrently,” Li says. That means our technology was stress-tested to handle, on average, about 9,000 users making trades at the same time. “But now all of a sudden we had to sustain 25,000 clients concurrently.” Engineers generally don’t feel comfortable unless their systems can seamlessly handle twice their normal “spike” capacity. But Wealthsimple’s systems were now facing three and four times that much activity on a regular basis.

Commission-free stock trading

WEALTHSIMPLE TRADE

Commission-free stock trading

Wealthsimple Trade makes it easy to buy and sell thousands of stocks and exchange-traded funds.

START TRADING

So how exactly do you scale a tech product that’s blowing up (in the best possible sense)? According to Li, there are a couple of ways. The first is vertical scaling: that just means going bigger. You increase the size of a server by getting a bigger machine, bigger database, faster processor, what have you. “It’s like a bus,” he says. “You go from a single-decker bus to a double-decker bus to maybe a triple-decker bus — it’s still taking up the same amount of physical space but you can move more people.” Throughout our run of growth in 2020, we did plenty of vertical scaling.

Sooner or later, though, you can’t make the bus any bigger. You need more buses. More hardware. This is horizontal scaling, and it’s what allowed us to keep pace when everything went bonkers during GameStop. “In a couple of days, we doubled the number of containers we were running behind the scenes, from approximately 1,000 to about 2,000,” says Li. “And when you do that, your bottlenecks become very apparent. We were able to see where things were getting held up, processes that were really slow. This allowed us to keep up with the ever-increasing number of concurrent users.”

“We were all-hands-on-deck in a virtual Zoom any engineer could join to watch and learn,” says Simoneau-Ritchie. “And when we were increasing capacity, a handful of our systems might need to go into maintenance mode for a few minutes so we'd plan to do that at night to minimize disruptions to clients.”

“It was very stressful,” Li says. “But also pretty exciting.”

And in the process of solving these problems, we solved other problems we didn’t even know we had. Prior to GameStop, our team mostly focused on ensuring quality performance for people simultaneously using the app. Now, for the first time, we started to see signs that people might be experiencing slowness when they were simply trying to sign up. When our engineers looked into it, they isolated a pain point we hadn’t noticed before: we were generating PDFs during the user sign-up process that could simply be generated later. And when we fixed that, we saw a significant uptick in performance. “Those types of optimizations have huge impacts on our ability to onboard new clients,” Li says. “As a result, our trading actually got faster during the GameStop period. Though, of course, there are still moments of slowness. We’re working on those, too.”

The whole thing reminded Li of a favourite aphorism of his: never waste a good crisis. “Some of our issues were long-standing problems that probably never would have gotten fixed if we did not have this crisis,” he says. “And when you go through stressful periods like that, the company and the engineering team learn a lot.”

Written By

Wealthsimple

Wealthsimple uses technology and smart, friendly humans to help you grow and manage your money. Invest, save, trade, and even do your taxes in a better, simpler way.