When payday is approaching, Q2 expects a 13 to 14 percent bump in users.

“We always have a plan to handle that,” Q2’s CIO and SVP Lou Senko said of the white-label online banking solutions provider.

What Q2 didn’t have a plan for was the flood of users following the federal stimulus check payouts in April.

“We anticipated an increase of about 30 percent heading into Monday, April 13 — double the normal peak-over-peak increase — when the first stimulus checks would hit people’s bank accounts,” Senko said. “By 6:30 a.m. Monday, we could see we were already above 50 percent.”

By Wednesday, volume had increased by 230 percent. And on Monday, April 20, when the second wave of stimulus checks dropped, user traffic peaked to unseen levels.

“We add about four users a minute to the platform, but no one on our team expected login activity to grow by 700 percent for some of the financial institutions we host,” Senko said.

As Q2’s network infrastructure and 10,000 servers neared capacity, the threat of online services — like digital deposits, money moving and lending — crashing for more than 140 of the 430 hosted financial institutions on the company’s platform intensified.
 

No one on our team expected login activity it to grow by 700 percent for some of the financial institutions we host.”

“All the online banking providers struggled to keep up with this unforeseen volume across the industry,” Senko said. 

But after members from its hosting, support and development teams acted swiftly, Q2 did keep up. Here’s how.

Monday, April 13

With a global team of 1,700 employees and 17 million users, Q2 no longer classifies as a startup. Still, it is empowered to take quick action and as Senko said, “We don’t have a lot layers to get approvals.” 

“We try to act like a small company by holding on to that startup feel — and that means moving fast,” Senko said. 

So at 11 a.m. on Monday, April 13, when federal stimulus checks began depositing into bank accounts and user traffic climbed beyond Q2’s expectations, Senko made a quick decision.

To keep pace with user traffic, Senko tapped into Q2’s reserve hosting solutions and reserve servers that he purchased at the start of the year to accommodate upcoming customer implementations.
 

I essentially broke the ‘in case of emergency’ glass.”

Overnight, Q2’s server capacity increased over 200 percent, readying for just under 20,000 servers. 

“We’re as lucky as we are good at having purchased the equipment upfront. Knowing that COVID-19 could potentially disrupt future supply chains, we made the decision to pull the year’s worth of purchasing forward,” Senko said.

For April’s situation, Senko enlisted the help from both the recovery equipment and hosting services supply for 2020, about $6 million worth of hardware total. Q2 also has a disaster recovery equipment supply, like servers, compute and storage, in case a facility or a piece of critical infrastructure goes down.

“In this situation, I essentially broke the ‘in case of emergency’ glass,” Senko said.

Reacting remotely

Once the hosting infrastructure team had the equipment deployed, the other teams quickly provisioned specific capacity into each customer’s stack — a task that would require the team’s first remote all-nighter. 

“Our 430 customers are like snowflakes; each one is different with its own customized solution, and about a third of them were impacted by this event,” Senko said. 

By tracking minute-by-minute login rates using internal data tooling and their sophisticated error monitoring, the team identified which customers were running normally and which ones were experiencing surges. From there, the team deployed the capacity, which took up the majority of the night. 

“Keeping a team motivated while working all night from our homes was a new challenge of its own,” Senko said. “Usually we’d be together, ordering pizza and pushing through.”

To adjust, Senko said he provided more verbal coaching than he usually would over calls, messaging and video conferences and stressed transparent communication amongst the team members, ensuring that the team knew what exactly their role was. Senko added that the team had a lot of help from the marketing, sales and customer experience teams to message back to the customers. 

“They’re experts,” Senko said. “It was an uncomfortable issue, but they knew what needed to get done, and then they did it. It’s moments like this where excellent teams stand out.”

Reflect and refill

Following Monday night’s efforts, Senko said he spent Tuesday personally checking in with customers to make sure their operations continued to run smoothly. Very few issues were reported, Senko said. 

On Wednesday, Q2 saw the 230 percent login activity spike, as the two-day automated clearing house (ACH) hold was removed from the checks and people could access and move the money. It was another tough day, Senko said, but the teams were able to carry through. 

“Deploying everything we had and pulling that all-nighter paid off,” Senko said. “We didn’t have a single capacity reported problem in the weeks after when the second and third waves of stimulus checks were sent out.”
 

Deploying everything we had and pulling that all-nighter paid off.”
 

That Friday, Senko met with Q2 executives for a debrief, highlighting the issues and the fixes through a five-page slide deck. His colleagues seemed impressed.

“My favorite part from this whole experience was sharing with my team what Q2’s president and CEO and CFO said in our debrief meeting about what an excellent job the team did,” Senko said. “It made it easier to ask our executive team to approve refilling our $6 million worth of equipment we just used.”