Test & Evaluation for Autonomous Military Vehicles: Challenges (Part 1)
Aug 10, 2022
The Department of Defense (DOD) is pursuing a wide range of autonomous and semi-autonomous systems for various defense applications. As the DOD and, in particular, the Army procures these systems, it is essential that safety and performance evaluations meaningfully address the broad spectrum of scenarios that the vehicle could encounter on deployment, without derailing program schedules.
This blog post is the first in a two-part series covering test and evaluation considerations, approaches, and solutions for autonomous military vehicles. Part 1 of the series focuses on how testing and evaluation challenges differ across autonomous vehicles built for commercial and defense applications, and why relying on real-world testing alone is insufficient.
In both military and commercial applications for autonomy, robust verification and validation processes are as critical to success as the development of the autonomy functionality itself.
Safety validation for commercial automated driving systems (ADS) and advanced driver-assistance systems (ADAS) is inherently difficult due to the unbounded conditions they need to navigate in their operational design domain (ODD). There are an infinite number of edge cases that an autonomous vehicle must be able to handle safely, presenting a serious challenge for engineering and testing teams.
Commercial autonomy is all about mobility: Vehicles need to stay on the road, inside the lines, respond to traffic signals, and avoid pedestrians and other vehicles. These are significant challenges, particularly when you consider how quickly situations can change.
Military vehicles inherit many of the same mobility challenges faced by their commercial counterparts and additional challenges that further complicate their deployment. For example, military vehicles must be able to navigate varied terrain at warfighting speed, often in communications-degraded environments. Military AVs must operate in environments that are less dense but also less structured than the environments faced by commercial AVs. Destinations and paths are not typically known - instead, a vehicle must position itself on the right route to optimize its ability to detect an adversary while minimizing the threat to itself.
Beyond mobility, certain military AV programs also have autonomy features built into survivability, fire control, and mission command functions. Integrating autonomy features into those functions further complicates safety and performance validation efforts before operational testing and, eventually, deployment.
Real-world testing of a vehicle’s performance in an unstructured, off-road environment requires access to large test ranges with vast expanses of varied terrain corresponding to the vehicle’s expected operational environment. Real-world testing also requires significant amounts of time to test and retest the autonomous system across enough edge case scenarios to validate its safety and performance.
A RAND study on commercial AVs, for example, revealed that a commercial AV would need more than 11 billion miles of testing to demonstrate 20% better safety than a human driver, based on a human driver failure rate of 1.09 fatalities per 100 million miles driven. Even under aggressive testing assumptions, a fleet of 100 AVs would take more than 500 years to reach that benchmark.
When considering the additional challenges associated with autonomous military vehicles, relying exclusively on real-world testing to demonstrate safety across the full spectrum of scenarios that vehicles might encounter in deployment is an impossible proposition. Defense autonomy programs can, however, augment real-world testing with virtual verification and validation processes to safely address coverage gaps left by real-world testing programs.
Get started with virtual testing: Learn more by reading our free V&V handbook, or request a demo to see how your team can use virtual modeling and simulation to prove out your autonomy stack.
.webp)
.webp)
