Apple's preliminary disengagement data for its self-driving car project surfaced yesterday pointing towards a high number of disengagements, and today, the DMV has shared the full disengagement reports from the company, providing more insight into Apple's autonomous car testing.
A disengagement report tracks the number of times an autonomous vehicle disengages and gives control back to a safety driver or the number of times the safety driver in the vehicle interferes.
Yesterday's information suggested Apple had the worst rank when it came to disengagements, but Apple has now provided details [PDF] explaining its disengagement reporting procedures and some changes that were made mid-year.
For the period between April 2017 and June 2018, Apple vehicles drove 24,604 miles autonomously and experienced 40,198 manual takeovers and 36,359 software disengagements, a number that is comparatively high based on disengagement reports from other companies.
As of July 2018, however, Apple stopped reporting its total number of disengagements and instead began focusing on "Important Disengagements," aka disengagements that might have resulted in a safety-related event (aka accident) or a violation of the rules of the road.
Using this metric, Apple vehicles have driven 56,135 miles since July 2018, with only 28 "Important Disengagements" reported. Two of these "Important Disengagements" were indeed minor collisions that weren't the fault of Apple's vehicles. One accident took place in August 2018 and the other was in October 2018.
Under Apple’s revised reporting threshold, the company’s cars experienced only one important disengagement every 2005 miles, compared to every 1.1 miles if the full data is counted. If other companies use similar thresholds to Apple’s new standard, Apple would rank much better.
Making direct comparisons between Apple's disengagement report and the results from other companies is difficult because there is no standard for reporting disengagements. It's up to each individual company to decide what constitutes a disengagement and what disengagements need to be reported.
It is clear, though, that Apple's vehicles are in the early stages of testing, as the company says itself in a DMV cover letter.
According to Apple, safety is its "highest priority" and its approach to disengagements is "conservative" because its system is not yet able to operate in "all conditions and situations."
Apple's testing parameters require drivers to proactively take manual control of a vehicle any time the system encounters a scenario beyond its current capabilities. The vehicle itself also self-monitors and returns control back to the driver when errors or issues are encountered.
Situations where drivers take over include the appearance of emergency vehicles, construction zones, or unexpected objects in the road, as Apple's vehicles cannot self-navigate these obstacles.
The autonomous software hands over control when it can't sufficiently track an object, is unable to generate a motion plan using the path planning system, when the vehicle systems don't respond as expected, and when there are communication issues.
Apple now has more than 62 vehicles out on the road, a number that will likely ramp up in 2019 as autonomous software testing continues. Apple is required to provide annual disengagement reports to the DMV, so we'll see the company's 2019 performance in early 2020, and will be able to look for improvements.
