Background
Autonomous Vehicles (AVs) hold the promise of substantially enhancing the safety and efficiency of our transportation systems, offering potential reductions in traffic accidents and improvements in traffic flow. However, despite these considerable advantages, widespread deployment of AVs remains limited. The reason for this is multifold. But above all, the safety performance of AVs is still below that of human drivers.
One bottleneck for enhancing AV safety performance lies in the safety testing and validation. This is a very challenging problem because of the rarity of safety-critical events in spatiotemporally complex driving environments. Most of the existing testing methods suffer from intolerable inefficiency and biasedness issues. This has severely hindered not only development and deployment but also the consumer trust of autonomous vehicles.
The main objective of the AV challenge is to rigorously assess the driving intelligence of AVs in realistic traffic environments. An interactive simulator is provided featuring a high-fidelity simulation environment within Mcity, a dedicated AV testing facility that incorporates a diverse range of driving scenarios, including highways, intersections, and roundabouts. Participants will be tasked with designing an AV decision-making module that enables the successful completion of various routes, prioritizing safety, efficiency, and comfort.
Challenge Description
To evaluate the driving intelligence of AV, the testing procedure is to deploy the user-developed AV agent (specifically the decision-making module) in the simulation environment, observe its performance based on various metrics, and make statistical comparisons. In contrast to conventional scenario-based testing processes, the Mcity AV challenge emphasizes creating an environment-level testing ground and poses the following uniqueness.
- Time-continuous testing environment: Unlike traditional scenario-based tests that focus on short, isolated events, the Mcity AV Challenge introduces a continuous testing loop within Mcity. This setup exposes the AV to a diverse array of background agents and situations over a longer duration, offering a comprehensive assessment of the AV’s capability to handle real-world, full-length trips.
- High-Fidelity Interaction: The challenge features an interactive testing environment where background traffic dynamically reacts to the AV’s decisions, enhancing the complexity and realism of the test environment. To ensure the simulation fidelity, the background vehicle driving behavior is calibrated using real-world data to reproduce realistic interactions. For safety-critical events, which are critical to AV safety, we validated that the crash rate and crash type generated in the simulation are consistent with the real world, using the last 10 years of crash data in Michigan. Such statistical-level accuracy ensures the trustworthiness of testing results. More information about simulation fidelity can be found in related studies [1].
- Efficient and Comprehensive Testing: By employing advanced accelerated testing algorithms developed in our previous studies [2, 3], the Mcity AV Challenge can achieve an efficient testing process without loss of evaluation unbiasedness. This efficiency is coupled with a broad evaluation framework that considers various performance metrics, including safety, efficiency, and comfort. This dual focus ensures not only an efficient assessment process but also a multifaceted understanding of an AV’s driving intelligence and operational capabilities.
These distinctive aspects of the Mcity AV Challenge underscore its advantages in advancing the evaluation of autonomous vehicles, offering a systematic, comprehensive, and realistic AV testing benchmark that promises to accelerate the development and deployment of safe, reliable AV technology.
Mcity Test Facility Overview
The Mcity Test Facility, located on a 32-acre site on the University of Michigan’s North Campus, features more than 16 acres of roads and traffic infrastructure. This full-scale outdoor laboratory is designed to simulate a wide range of complexities that vehicles face in both urban and suburban environments. It serves not only as a platform for vehicle testing but also provides the connected infrastructure and operating system necessary for a smart city test bed.
Mcity Simulation Environment
For the AV Challenge, we’ve developed an interactive simulator that can generate a high-fidelity traffic environment in Mcity to test participants’ decision-making modules. This simulation environment features:
- Diverse Road Types: Including signalized intersections, urban roads, roundabouts, and highways, to assess vehicle performance under varied conditions.
- Interactive Simulation: With background vehicles that interact with each other and the AV being tested.
- Realistic Safety-critical Events: Based on a decade of crash data from Michigan, USA, providing accurate crash rate and crash type distributions.
- Diverse Traffic Demands: To simulate real-world traffic conditions with different traffic demands.
Updated on Mar. 20, 2024. Version 1.2