The late Sixties — and 1969 in particular — were a time of turmoil and change. The Beatles performed their last public concert. The Vietnam War did not de-escalate. Neil Armstrong landed on the moon. Nixon was sworn in. Woodstock brought over 400,000 attendees to Max Yasgur’s dairy farm.
And in February of that same year, the new 747 — the world’s first jumbo jet and self-proclaimed “Queen of the Skies” — made a maiden voyage over Seattle that many thought impossible. Getting the bird off the runway, however, was no easy task.
“I saw Boeing’s new jet as 75,000 drawings, 4.5 million parts, 136 miles of electrical wiring, 5 landing gear legs, 4 hydraulic systems, and 10 million labor hours,” wrote renowned engineer Joe Sutter.
Half a football field long, wide as a 20-lane highway and requiring 184 mph under standard conditions to lift off the runway, making the 747 fly was an enormous technical challenge. And Sutter, considered the “father of the 747,” knew that critics were just as dubious about sticking the landing. “How could pilots judge the landings, these critics said, when the cockpit was three stories off the ground?”
Yet after takeoff and flight — and because of those 10 million labor hours — the landing appeared luxurious: “Before my eyes,” Sutter wrote, “it descended to the runway with the stately majesty of an ocean liner.”
The good news is that runways aren’t only reserved for 747s. In the tech world today, many companies, teams and leaders seek to break new ground by transcending their technical challenges and sending new projects far into the stratosphere.
Built In Austin sat down with senior software engineers from Apptronik and Validere to hear about their biggest technical hurdles to date — and how they turned their professional roadblocks into career-defining runways.
Apptronik is building a robot to work alongside humans, to walk like humans do and to manipulate objects with its hands. The company’s goal is to build robots for the real world to improve human quality of life and to help solve the ever-increasing labor shortage problem.
What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?
As a senior software engineer working with humanoid robots, one of the biggest technical challenges I have faced recently is integrating various robot actions to form increasingly complex behaviors. The humanoid robot requires integrating solutions from experts in multiple domains, including control, planning, and perception. This can be a tricky challenge because it involves a lot of coordination and collaboration from experts in those fields.
We typically use state machine architecture, but as the robot’s skills grew, my team and I found that the tasks for the robot became more complex and increasingly difficult to ensure that the state machines were functioning correctly and robustly — a real head-scratcher! In addition, it required a lot of time to ensure that it could perform the given application and modify it for different applications.
How did you and your team overcome this challenge in the end? What were some of the specifics tools or technologies you used?
In order to overcome this challenge, we conducted research and found that the video game industry had been using a technique called behavior trees for AI Characters. This technique has also become popular in robotics due to its increased scalability, flexibility, modularity, and ability to perform reactive actions and more expressiveness than the state machine.
After evaluating different options, we decided to combine multiple approaches in a hierarchy and implement behavior trees for high-level decision-making and low-level control with a finite state machine in our robot software. This made it easier for the robot to combine multiple skills for its application. Additionally, we also used tools such as simulation software and testing platforms to test and evaluate the robot’s performance with a new approach.
In conclusion, by using the behavior tree technique and combining it with a finite state machine, we were able to overcome the technical challenge associated with state machines and improve the level of intelligence, agility, and flexibility in our robot. This approach has been a game-changer for our team and has allowed us to develop more advanced and capable robots.
By using the behavior tree technique and combining it with a finite state machine, we were able to overcome the technical challenge.”
How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?
This technical challenge of integrating various skills to form increasingly complex behaviors was an excellent opportunity for me to strengthen my problem-solving skills, as well as my knowledge of AI and robotics.
Additionally, it helped me to gain experience in implementing new technology and making decisions that would have the most positive impact on the project. It also helped me to strengthen my communication and collaboration skills as I worked closely with experts from each robotics domain to develop a successful solution and integration with the new architecture.
It is also an excellent reminder that, as software engineers, we must always be willing to adapt and learn. We must be open to new technologies and be ready to learn and implement them to achieve our solution.
Validere’s mission is to better human prosperity by making the energy supply chain efficient and sustainable. The company provides energy organizations with clarity into their physical and environmental commodities, through technology and expert guidance that helps them trust their data, zoom in on important insights and act on them with ease.
What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?
We have been working on a system that allows us to raise alarms if specified incoming data streams have user-specified anomalies. An example of this is if a sensor reports that a pipe has a pressure that falls below a specified threshold for a certain amount of time or if it rises above a different threshold at all. This is tricky because we need to monitor multiple streams of data at a time in order to generate our alerts, and the queries to obtain enough historical data to evaluate alarm conditions could be very costly to execute both in terms of execution time and data transfer. Therefore, we need to preserve a state of every alarm that we need to evaluate and perform error recovery in such a way that does not require us to rebuild state if there is an error, missing data or deployment.
How did you and your team overcome this challenge in the end? What were some of the specifics tools or technologies you used?
While the functional requirements of the system were well known, the nonfunctional requirements are the ones that drove the design of the system and informed the choice of tools that we should use. As we want to use the right tool for the job, we chose to build this out in Elixir, not only because we are familiar with it, but because Elixir allows us to build out such a solution in a fault-tolerant manner so that if any alarm experiences an issue, it does not bring the system down or require us to rebuild state. We are able to maintain alarm state in lightweight processes so that every time new data is received, we do not need to query for the rest of the data in the evaluation time window. This allows us to quickly process new alerts.
We also cache the state in a durable cache, so that if there is an error that causes a crash or a deployment that tears down all of the existing processes, we can rehydrate the system with the last known states in a very efficient manner and resume processing data again. Furthermore, our processes are networked together so that we can ensure there is one monitor process for every alarm so that we don’t raise duplicate alarms.
We are able to maintain alarm state in lightweight processes so that every time new data is received, we do not need to query for the rest of the data in the evaluation time window.”
How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?
This challenge helped me grow as an engineer because it challenged me to use all of the tools that I have used in the past, in ways where I have not used them together. It also made me evaluate my initial instincts to understand if that tool would fully enable meeting the customers’ needs. This critical thinking experience is vital to all of us engineers because we can very easily paint ourselves into a corner when choosing certain tools, but we have designed this system in such a way that we can easily swap out some dependencies and strategies after receiving more feedback from stakeholders. During the course of building out features we find that there is a dichotomy between meeting the current need and being able to meet future needs. This challenge forced us to continually evaluate our position in this dichotomy so that we could build with our current goal in mind, without restricting or limiting necessary future enhancements.
