In recent interviews with four local engineers, Built In Austin asked them about a problem they all dealt with.
Their response? They figured out, in one way or many ways, a path to solve it.
And while that might seem overwhelmingly obvious, the four also reflected on what they learned from their challenges — the biggest highlight of the problem-solving process.
“While we, as a team, certainly believe in the sentiment that you shouldn’t reinvent the wheel, there are certainly times where it makes sense,” Loomy’s Engineering Lead Ari Summer said.
“Although I was aware of these concepts before working on this feature, this opportunity allowed me to fully understand the implementation of these concepts in a real-world scenario,” AdAction’s Senior Full Stack Developer Dusty Christy said.
“This challenge gave me a chance to grow my skillset rapidly in a way that I wouldn’t have been able to at a larger company like Facebook or Google,” REX’s Software Engineer Colt McNealy said.
“The project itself helped my team and I gain more practical experience in coordinating cross-team efforts and appreciation for the challenges they create,” ThousandEyes Engineering Leader John Shields said.
Below, the four local engineering leaders go into more detail about their problem at hand, the solutions they uncovered and the lessons they learned.
Summer, an engineering lead at the brand success platform Loomly, led us through how his team solved for reliably serving their static assets during a rolling deploy, a process Summer said can be difficult to navigate.
What’s the biggest challenge you’ve faced recently in your work?
One tricky challenge we faced was related to reliably serving our static assets (JavaScript, CSS, images, etc.) for our Ruby on Rails app during a rolling deploy process. Rolling deploys are tricky because, during the deploy process, you are temporarily and simultaneously serving traffic from both old and new versions of your app as machines are gradually updated with the new version. If you’re not careful, it can lead to unexpected consequences when deploying updates.
When it comes to serving static assets during a rolling deploy, you need to make sure to serve both the old and new assets during the deploy process since a client could be requesting either during the deploy.
How did you and your team overcome this challenge in the end?
One way to solve this problem is to maintain old and new versions of assets on a CDN (content delivery network). We use Cloudfront, backed by S3. In addition to removing the burden of maintaining old and new versions of assets on your machines serving app traffic, it reduces the load to those machines and provides edge caching for faster load time for our users.
In order to get our compiled assets on to S3 to be served via Cloudfront, we started using asset_sync, a Ruby gem that integrates with Rails to upload your assets to S3. We added this to our CI pipeline to upload assets before we started the rolling deploy process. This worked well for some time, but asset_sync didn’t provide an easy way to customize the configuration for our different environments (development, staging, production) and it also didn’t provide a way to delete old, unneeded assets from S3, allowing old assets to pile up in our S3 bucket.
We ended up taking inspiration from asset_sync and building our own library for uploading and maintaining our static assets on S3. This has allowed us to easily upload to different buckets for our different environments and to easily retire old assets after a configured amount of time. Thinking this could be useful to others, we have started to extract our work into an open-source gem called S3AssetDeploy.
There are certainly times where it makes sense to build your own solution for your use case if what’s out there doesn’t quite fit your situation.”
How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?
While we, as a team, certainly believe in the sentiment that you shouldn’t “reinvent the wheel,” there are certainly times where it makes sense to build your own solution for your use case if what’s out there doesn’t quite fit your situation. The hard part is choosing when this makes sense. Doing so gives you the freedom to gear your solution to your needs, but comes with the responsibility of maintenance and upkeep. It’s important to have your needs clearly defined before diving into a custom solution and working with what’s already out there can really help in providing some of that clarity.
Shields, an engineering leader at network intelligence company ThousandEyes, led us through the practical and technical solutions he and his team implemented for a migration that required zero downtime for users.
What’s the biggest technical challenge you’ve faced recently in your work?
We were recently migrating our primary customer-facing web application and API from an in-house data center to AWS. Since this application is high volume and directly customer-facing, we were required to perform the switch with zero downtime and no lost user sessions. The application is deployed on Kubernetes, so we were able to have the cluster span both data centers to allow the same application deployments to be available in both data centers. The tricky part was performing the network switch and moving from an internal F5 load balancer to an AWS application load balancer. This also required our canary mechanism to change, which also needed to be verified.
The technical challenges were interesting but the part that made it particularly tricky was the combination of zero downtime and the coordination of multiple teams within the company.
How did you and your team overcome this challenge in the end?
In the end, the way we overcame the challenges in this migration was part technical and part practical.
On the technical side, we were able to leverage AWS peering to provide a single Kubernetes cluster across both data centers. This allowed us to use the same deployment for each application, which made it easy to maintain sessions and ease deployment complexity. We also leveraged multiple ingress controllers in the Kubernetes cluster to support different canary mechanisms for the F5 traffic versus the AWS ALB traffic. Lastly, we utilized a temporary DNS configuration to allow us to fully test the new AWS load balancer prior to the switch. All of these (and other) technical approaches allowed us to make these production changes with confidence and with the ability to easily rollback if needed.
On the practical side, this effort required much coordination among our SREs, network engineers, application developers and engineering managers. We were able to create helpful project plans and runbooks for performing the migration. We utilized these written documents to align all of these groups and ensure everyone knew their roles and responsibilities.
The project itself helped my team and I gain more practical experience in coordinating cross-team efforts and appreciation for the challenges they create.”
How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?
My team was responsible for the overall coordination along with the technical aspects of the application deployment, new ingress configuration and new canary support. This effort allowed my team and I to gain a deeper knowledge of the networking details of both Kubernetes and AWS. We also went into much detail around various canary mechanisms and understanding how to leverage HTTP standards to make this work with different network topologies.
The project itself helped my team and I gain more practical experience in coordinating cross-team efforts and appreciation for the challenges they create.
As a member of the infrastructure/DevOps team at digital platform and real estate brokerage REX, McNealy and co. attempt to solve macro problems that will affect other engineering teams in the future. His most recent challenge? Ensuring the interactions between all of their mircoservices run smoothly.
What’s the biggest technical challenge you’ve faced recently in your work?
We have hundreds of microservices to support various capabilities, such as our incredibly high-performing and customized digital advertising, extensive back-office automation, data science and machine learning initiatives, and our rapidly-evolving yet elegant and friendly web and mobile app user interfaces.
Given how rapidly our tech stack is growing, it has become increasingly difficult to manage the interactions between all of our microservices. For example, how can I do A/B testing on a new API without changing the code of every caller? How can I determine exactly who uses microservice X? When a certain feature of the website is down, which of our several hundred microservices is the problem?
How are you and your team overcoming these challenges?
We are implementing a new architecture called a “Service Mesh.” Leveraging the Istio project, we include a proxy container on every one of our Kubernetes pods (i.e. microservices) which listens to, hijacks and intelligently routes all traffic that goes into and out of each REX microservice. This provides great visibility into how the system behaves.
This challenge gave me a chance to grow my skillset rapidly in a way that I wouldn’t have been able to at a larger company like Facebook or Google.”
How has this technical challenge help you grow as an engineer or help you strengthen a specific skill?
Through the process of making significant in-house modifications to Istio, I got a crash-course in modern distributed systems, networking and microservice architectures. Our company has much more work to do than people to do it, which means that I was given the opportunity to work on a new project despite having almost no prior knowledge of the subjects. This challenge gave me a chance to grow my skillset rapidly in a way that I wouldn’t have been able to at a larger company like Facebook or Google.
Research (and a lot of it) was necessary to determine how Christy, a senior full stack developer at mobile app media company AdAction, would implement an authentication service for the company’s API: open-source, paying for a service or creating their own in-house solution.
What’s the biggest technical challenge you’ve faced recently in your work?
When I started at AdAction, our tech stack was in its infancy and one of the first challenges we faced was creating an auth service to secure our APIs. We were faced with many challenges, like why not just use an existing open-source solution like KeyCloak, or pay for a service like Okta instead of rolling our own? This particular challenge was tricky because, although you can easily deploy a KeyCloak instance or sign up and pay for Okta, they both also come with their own set of negatives. KeyCloak had a lot of bloat that we wouldn’t utilize, and although Okta is feature-rich and has everything we would ever need, it’s also rather expensive.
We felt rolling our own service would be advantageous by giving us the freedom to add features over time as they’re needed, like social logins, and by being a cheaper solution in the long term.
How did you and your team overcome this challenge in the end?
We spent some time researching our three primary options: using KeyCloak, paying for Okta, or hand-rolling our own auth microservice. After various discussions, we decided hand-rolling our own service would be cheaper in the long run and provide us with the flexibility we wanted as our tech products continued to grow.
Our tech stack primarily relies on Spring Boot in the back end. We were able to wire up our auth microservice utilizing Spring Security. Although tackling auth can be an overwhelming task, Spring Security combined with Vault and Netflix Zuul gives us the confidence that our APIs are protected and secure.
These types of challenges are why I’m enjoying working for a smaller company.”
How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?
These types of challenges are why I’m enjoying working for a smaller company. My experience at larger companies leads me to believe an auth service will generally be something that was written a long time ago, or they will be utilizing something like KeyCloak, so I find it’s a decently rare opportunity to get to work on something like this.
This particular challenge gave me the ability to learn a lot more about Spring Security and how authentication is created, how user context is created via a JWT and how applications handle authorizing API calls. Although I was aware of these concepts before working on this feature, this opportunity allowed me to fully understand the implementation of these concepts in a real-world scenario.
