Imagine you’re driving down a road with your friend in the front seat. A child suddenly chases a ball into the middle of road, and your only choices are to run the child over or to swerve into a tree, which could kill you and your passenger. What do you do?
It sounds like a test to determine someone’s level of psychopathy, but it’s actually a choice the developers of autonomous vehicles have been faced with since the idea first materialised.
In an interview published last week with Car and Driver, the manager of driver-assistance systems at Mercedes-Benz, Christoph von Hugo, revealed that the company’s future autonomous vehicles would always put the driver first. In other words, in the above dilemma, they will be programmed to run over the child every time.
While either choice by the car’s developers may feel uncomfortable, it would be more dangerous if they didn’t address such a situation, because unless a self-driving vehicle is told what to do when a child runs into the road, it won’t do anything.
Manufacturers, however, had been quiet about what would happen under these circumstances until this month. Speaking at the Paris Auto Show, von Hugo told Car and Driver that all of the company’s future Level 4 and Level 5 self-driving cars would be programmed with the decision to save the people they carry over anything else.
"If you know you can save at least one person, at least save that one. Save the one in the car," von Hugo said, according to Car and Driver. "If all you know for sure is that one death can be prevented, then that's your first priority."
But does this viewpoint line up with the that of potential buyers? A study published in the journal Science this year highlighted the ethical dilemmas for those manufacturing self-driving cars and what survey respondents thought would be the correct course of action: kill or be killed.
Just under 2,000 people were polled, and most believed that autonomous cars should always make the decision to cause the least number of fatalities. On the other hand, most people also said they would buy one only if it meant their safety was a priority. That suggests that, when it comes to selling cars, at least, Mercedes has probably made the right call.
"According to our results, this might help sales, because what we observe is that people largely prefer to ride in a car that will save them at all costs, even though that is not their moral preference," Jean-François Bonnefon, a coauthor of the paper, told Business Insider. "So not only could that help sales, but it could actually make it difficult for other companies to program their cars otherwise."
At the show, von Hugo justified the decision by saying that if the lives of the people inside the car were uncertain, the subsequent effects would also be unknown. For example, thinking back the previous situation, if you had swerved and hit the tree, the child might still have been killed by a falling branch. Or maybe you cause a secondary collision with a school bus behind you that's holding 30 children, who all die. Maybe someone else runs the child over anyway.
"We believe this ethical question won't be as relevant as people believe today," von Hugo told Car and Driver. "It will occur much less often. There are situations that today's driver can't handle that - from the physical standpoint - we can't prevent today and automated vehicles can't prevent, either. [The self-driving car] will just be far better than the average driver."
Bonnefon said Mercedes was brave to bring the discussion out into the open. It's also a clever move from the company. Last month the US Department of Transportation issued 15 criteria that must be met before autonomous vehicles could use the roads. One requirement was ensuring that the programming could deal with ethical choices.
"It's fantastic that Mercedes-Benz has chosen to speak openly about this issue, because other companies have been silent," Bonnefon said. "At this stage we're sort of in the Wild West of self-driving cars' ethics, and it seems like in this case Mercedes drew first."
If a fatal accident does occur as a result of this decision, though, Bonnefon said, the backlash would still be huge. It's up to the individual manufacturers whether they want to take on that responsibility.
"So is that going to set the tone? That's very hard to say," he said. "Is it going to provoke government regulations? We'll see."