In software engineering, we're reaping the benefits of early investments in machine learning infrastructure, getting fast, detailed feedback from cutting-edge simulation tests, and building out key teams from Site Reliability Engineering to Developer Infrastructure. And this is still just the beginning. As we continue to grow, we’re thinking deeply about how we can support the thousands of engineers we’ll hire down the road.
As many of us in the industry know, scaling gracefully is a balancing act. We keep our standards high by staying away from demo-driven development, following style guides, and writing code that other engineers can understand and maintain. It’s equally important to actively prevent gatekeeping and bureaucracy. We make a point of hiring experts, and we win when we allow them to creatively apply their knowledge throughout the stack.
So, to help us do more of what works (and less of what doesn’t), we’ve collectively developed five Software Engineering Tenets that guide us day-to-day. Below, we share how they enable innovation, help us make good decisions, and generally make being an engineer at Aurora awesome.
#LowFriction: We enable engineers to rapidly experiment with our data sets.
To solve the complex problems we encounter in self-driving, our team must continually develop and test novel approaches. We prioritize removing barriers to this type of critical experimentation and provide environments where engineers can learn fast and iterate quickly. This is how we accelerate developer velocity and leverage our team’s deep expertise.
Jimmy Jin | Motion Planning - Machine Learning | PAO
“We will succeed by building an architecture that allows us to experiment faster than everyone else. This translates to two dogmas: 1) drive development with experimental results and 2) invest in experiment infrastructure. Every new decision model we create in motion planning is accompanied by a detailed analysis of new findings.
Ranging from improving robustness of our data generation pipelines, developing frameworks for tracking experiments at scale, to researching new evaluation metrics that bring clarity to model performance, the planning team has been putting emphasis on extracting the maximum amount of information from running each experiment. As a result, we’ve continued to make significant improvements towards highway driving capabilities, even when we had limited resources to run physical tests during the pandemic. “
Becky Abramowitz | Motion Planning - Decision Making | SF
“#LowFriction acknowledges that in order to move quickly, we need to be able to run experiments with urgency and at scale. This includes rapidly iterating against our large simulation suite, running data analysis in Python, and closing the loop by working with Vehicle Operations to gather real world data.
Not only does the emphasis on experimentation help us move faster, it also makes being a software engineer here all the more exciting and fun. I think it's especially awesome that, despite joining as a recent grad, I get to be on the research side and feel like I'm truly helping to build the Aurora Driver in a meaningful way.”
#AdviceNotGateKeeping: We believe in a culture of “Yes, and” instead of “No, I know better.”
Creating excess layers of approval prevents important experiments from occurring. We do encourage teams to seek advice from experts, but experts at Aurora don’t block a course of action unless it contravenes a company policy. Trusting teams to use their best judgment means allowing the occasional mistake, but this ultimately produces better engineers.
Fiona Barry | Developer Infrastructure | PIT
“I work on maintaining our build server and our cloud infrastructure. These are tools that everyone has to use, so I’m often pulled into code reviews. If a part of a code review is outside my domain of expertise, I don’t block the review. Instead, I ask for a high-level explanation and often pull in another reviewer (from the author's team) with more relevant knowledge. Because there’s no gatekeeper, the submitter and the reviewer feel like collaborators, not opponents. This works because of the mutual trust and respect developers here have for one another–we don’t do finger-pointing when the occasional mistake is made.
Another really cool result of this mindset is that anyone at Aurora can write a design document, and if it makes sense and improves the productivity of the company, people will review it and take it seriously. This allows developers to take ownership of their work and make meaningful contributions in ways they might not be able to elsewhere.”
#CarryOurWater: We’re all responsible for building and maintaining our tools.
Every team must do their part to keep our systems nimble, effective, and scalable. At times, it makes sense for people with specialized skills to do more “water carrying,” but we never expect them to do so indefinitely. To effectively spread out the responsibility, we first ensure that we invest in documentation and API quality for all of the tools we build. Second, we don’t have a separate research team. We hire both high-quality researchers and world-class software engineers who work in tandem across the stack.
John Harrington | UI Tools | PAO
“Any time I’m introduced to some new technology required by a new project, I try to dive as deep as possible so that I can handle the initial integration into our project and also understand it enough to ensure its maintenance over time. For example, when we transitioned a labeling tool to a new environment, I had to become familiar with Kubernetes, Docker, Nginx, and Bazel–all technologies I’d never personally used before.
It essentially comes down to personal and team responsibility. As an engineer, I’m responsible for the code I write, and as a team, we’re responsible for our applications. We’re empowered to troubleshoot issues ourselves, allowing us to avoid being blocked by cross-team dependencies.“
Traian Stancescu | Simulation | SF
“Simulation is one of the bigger software teams at the company, and when we need a new feature or tool, we don't ask others to do it for us. We look at how similar problems have been tackled in the past, ask for guidance, and ultimately get at it ourselves. This often reduces turnaround time significantly– sometimes from weeks to days–which makes Simulation engineers happy. We avoid adding extra load on other teams, and improve the quality of the final product.
There is one essential requirement to make this tenant work, and that is effective cross-team communication. We collaborate tightly with the teams that have experience in building the tools we want, and often have them involved in the development cycle for advice. #CarryOurWater goes hand in hand with #AdviceNotGatekeeping, which is also very dear to me–it would be hard to have one without the other.”
#EndToEnd: We’re thoughtful about when and how we build custom solutions.
We believe that good > perfect, meaning we’ll accept an “imperfect tool” that comes supported “out of the box” until it truly makes sense to develop a tailored solution. When we do build a custom tool, we build it end-to-end before transforming it into a framework. This allows us to discover requirements we missed in our up-front design and get a better sense of what we truly need to move work forward. Finally, we avoid demo-driven development or the urge to productionize proofs-of-concept. We do the work to build a good tool and then commit to supporting what we build.
Kevin Allen | Teleassist | PIT
“We don’t start working on new software until we understand how it will fit into the Aurora Driver. This is a huge motivator for me, as I know that every contribution I make will be instrumental for our autonomy system.
I recall seeing the power of End-to-End when I started work on a part of Teleassist that helps the Aurora Driver handle situations like detours and construction zones. Rather than being given some small part of this task to work on in isolation (like a rigidly spec'd library, as I have had in the past), I met with engineers on our Motion Planning, Networking, and UI-tools teams to understand the other software components that my project will ultimately interface with. I met with my tech lead and product manager to understand the necessity of my work to our long-term vision. I met with experts in vehicle operations and simulation to understand how this feature can be rigorously tested. Then, clear in exactly how this feature fit into our system, end-to-end, I set about designing the best solution.”
#FirstClassLanguages: We are a multi-language company.
We are a company that uses several languages, including C++, Python, and flavors of JavaScript. But we don’t lean on language experts to keep our codebase accessible–we enforce best practices and style guides. When new languages enter the company, we ensure that the Aurora ecosystem evolves around the language as the maturity of its use evolves.
Craig Sebenik | Site Reliability Engineering | PAO
“We’re continually improving our tools and our infrastructure to make it easy for both casual and experienced Python programmers to use the language without sweating the details. First, we have a team of people working on libraries to make everyday tasks easier for other engineers. Second, we have build tooling to make it easy for Python developers to build, test, and distribute their software. Finally, we have an amazing internal education emphasis. Many of our training tools (codelabs) have Python examples, and we recently completed an "Intro to Programming" internal course that was all taught in Python. ”
Ryan McDougall | Infrastructure | PAO
“Aurora stands out because of how we encourage and reward individuals like myself to take ownership of code quality. First, the company values my more than 15 years of experience in C++ for real time systems, and funds my participation in C++ International Standardization Working Group, for which I’m currently assistant chair of SG21. Second, Aurora supports self-organized internal education initiatives to level-up internal skills. I recently designed and taught a class that helped other engineers feel more confident tackling difficult C++ development.”
