Safety: Our metric for success in autonomous vehicle development

With the commercialization of autonomous vehicles frequently referred to as a “race,” the discussion of competitiveness often turns to which company holds the best technological advantage over the others.

However, while Torc is notorious among internal teams for setting aggressive technological goals, we also have the perspective of over a decade of commercializing self-driving technology for a variety of different industries. Our approach is an iterative one, focused on safety.

We recognize that on the road to commercialization, it is the team that can build a sustainable, safe solution that will ultimately deliver. Under the pressures of growth, safety becomes more critical, not less.

Earlier this year, like many organizations in growth mode, our management team unveiled a set of company-wide goals for bolstering infrastructure, releasing new software capabilities, and increasing our fleet efficiency. On top of these goals came the overarching mission presented by our CEO, Michael Fleming: a safety goal to focus all teams on working to preventing incidents.

Safety as a metric for a successful test

To our Operational Safety team, a safe test begins with preparation and is the key indicator of success.

Kyle Lansing, Operational Safety Engineer
Explain safety as a metric in testing.

I recently trained a new group of Safety Conductors and I explained the importance of why every member of the test crew should strive for tests that are safe, efficient, and successful. Safety is the key metric there. Any test, in order to be successful, must be safe. For example, say a test doesn’t provide all the data we were originally looking for. If that test is executed safely, then it was successful.

What is a safe test?

Obviously, it is first a test that avoids danger or risk. But I believe it’s more than just everyone coming back having avoided a crash. If our test crew wasn’t prepared or if someone was uncomfortable with the vehicle behavior, Torc wouldn’t qualify that as a safe test.

How do you incorporate safety into your day-to-day job?

While we train and prepare to act rapidly in critical situations, I believe the true reflection of a robust safety program lies in the day-to-day operations. Every well-prepared test plan, checklist, and brief is one step toward ensuring a safer test.

In my day-to-day life, a safe test begins about 1.5 hours prior to us ever starting up a vehicle. There are inspections, crew task lists, and an assembly about the goals of our trip and expected behavior from both the vehicle and the test crew. Safety starts with preparation and I work to make sure everyone is prepared.

Myra Blanco, Daimler Torc Senior Technical Fellow
Explain safety as a metric in testing.

An important measurement of whether an effort is safe is the absence of safety-related incidents (e.g., crash, near crash). However, if we only look at those incidents that result in severe consequences – crashes – it might be too late. There are probably a lot of things we could have caught beforehand (e.g., near crashes, minor protocol discrepancies). Incidents are symptoms of actions that were not appropriate before the actual issue manifests. Defining safety metrics involves making sure you have put all the possible safeguards in place.

Torc software engineers rigorously test code prior to on-road testing.

There is not a single metric of safety, it is a conglomerate of multiple things. If we take crashes, for example – I don’t call them accidents for a reason – crashes have contributing factors. Let’s look at this metaphor: If you think about a piece of swiss cheese that has a lot of holes, imagine each one of those holes is a factor. When all those holes intersect, something can fall through the middle – this is akin to when multiple contributing factors line up to cause something to go wrong. So, it is not an accident that an incident happens – it is a contribution of multiple issues. In the case of a typical car crash, perhaps the person was distracted, was fatigued, or was speeding, combined with the vehicle having a potential fault or failure. None of those things happen randomly.

To help prevent these issues, my team works to identify and mitigate contributing factors at four levels: people, technology, environment, and vehicle. For example, a metric of success is ensuring you have 100% of your drivers trained. This shows that Torc is providing the appropriate information to do their job safely. Another would be ensuring that there are appropriate communication systems in place. Communication channels that are open are one of the most important parts of developing a safety management system.

Safety starts long before the road

Clearly, the process of avoiding incidents begins long before we put tires to the road. This process is established in the technical development stages as well, as our software engineers explain below.

Manas Gupta, Software Engineer in Behaviors, Planning & Controls
What is one task or process that you perform that forwards safety in your day-to-day work?

Safety is the most important and integral part of all the core values at Torc and we as a team work to develop software without compromising those core values. We make sure that our self-driving software is passed through various stages of testing before it is ready to deploy on the vehicle and can be run on public roads.

I follow three stages of testing to make sure the software I write or modify is safe to be deployed on the vehicle. First, I safeguard my code with unit tests for the given functionality, so that future unwanted changes can be caught in the development. Second, I seek peer review of my code. Last, I test the code on a simulated environment with all the possible scenarios to make sure the code is robust and can be tested on the autonomous vehicle on public roads.

Elijah Hodges, System Integration Engineer Co-op in Systems Integration
Explain safety as a metric in testing.

The Torc mission is to save lives. Safety is not just a part of our goal, it is our goal. If the tests we perform were to introduce dangerous situations, then we are going directly against our mission.

How do you incorporate safety into your day-to-day job?

The System Integration team is responsible for overseeing the high-level picture of vehicle performance. We help coordinate releasing new development onto our self-driving vehicles for testing to ensure that these changes improve the system as a whole. While a developer may just be implementing a change to fix a very specific issue, it is our job to notice and comment on how the change may affect peripheral responses. Ultimately, we are in charge of deciding if new changes are safe enough to be accepted into the software. Because we are not writing the code ourselves, it takes away the emotional temptation to accept new code changes just because they work and encourages us to determine if the overall performance was improved.

One day-to-day task that improves safety is our morning sync meetings every morning. Communication is absolutely vital for safety and having a venue for promoting that every single day is one of the best ways to find problems before they escalate.

When humans might fail, process prevails

The string that has tied all these different teams together is process. Process helps us make sure the important tasks identified are accomplished every time.

Denise Thompson Harmon, Department Manager in Behaviors, Planning & Controls
What is one task or process that you perform that forwards safety in your day-to-day work?

Being an Agile coach, I have always been driven by process. We humans are fallible, and we are going to forget to do things. So, my goal is to make sure that we have trained well enough and ingrained these processes so that they become a habit – it’s just what we do. My goal every day is to make sure we are being clear on our processes and keep training individuals in those processes to ensure that they are following them.

Additionally, we look for opportunities to grow. One of the agile pillars is relentless process improvement. With technology changing so quickly, and industry changing so fast, we can’t let our guard down. So, every day you have to be learning, every day you have to be growing, and every day you have to be focused. We know that humans are fallible, so we ask ourselves, “does our process support catching human error before we deploy software into the field?” I want to make sure we build that quality into the product and it is not an afterthought.

Safety culture involves everyone

While process has been interwoven into all these perspectives, so has the importance of communication. In that way, our mission is also reinforced by our safety culture.

Ruel Faruque, Sr. Operational Safety Engineer
What does your team do to contribute to the safety culture at Torc?

When put in charge of Operational Safety for Torc’s self-driving passenger car project, I learned how important it was to make safety a dialogue – to have discussions with the team carrying out the policies and to witness the protocols in action. Safety policies and protocols are much more effective if the team following them understands the underlying motivation, and many times, the team can suggest a more efficient and robust solution to the original safety concern.

Torc’s safety culture centers around three principles:

1. Speak up – if you have a safety concern, voice it. We provide multiple methods to do so, whether it’s immediately to the test crew, to your team lead, or via anonymous feedback.
2. Lessons learned – Learning lessons is a mission of continual improvement. At the end of a test or any other effort, it’s important to honestly evaluate what went well, what could have gone better, what was learned, and what should be changed for the future.
3. Focus on preventing future incidents, rather than identifying singular blame. Rarely does an incident have a single cause. Identifying all contributing factors yields multiple areas to make corrective action [summarized from Engineering a Safer World by Nancy Leveson].

The Operational Safety team teaches people the safety culture, empowers Torc’rs to exercise it, and keeps a pulse on the effectiveness, adjusting as needed to keep safety a priority at Torc.