Markus Maurer
Technical University of Braunschweig
Institute of Control Engineering
Ethical aspects of decision-making have been at the focus of increased discussion over recent years, as they play an important role in the social acceptance of automated vehicles above and beyond consideration of mere technological aspects. Whilst public discussion has centered above all on situations of dilemma involving unavoidable personal injury, the “Value-based decision-making” (VBDM) project sponsored since 2016 by the Daimler and Benz Foundation considers the influence of ethical issues on everyday decision-making in road traffic. The project is headed by Prof. Dr. Markus Maurer from the Institute of Control Engineering at the Technical University of Braunschweig. The main participants in this research team are Susanne Ernst, Marcus Nolte, Dr. Andreas Reschka, Nayel Fabian Salem, and Torben Stolte. In order to stimulate and intensify interdisciplinary dialogue between the engineering sciences and ethical issues, the scientists are working together with partners from the USA and Canada. The project has been receiving significant research impulses in an exchange of ideas with Prof. Chris Gerdes and Sarah Thornton from the Dynamic Design Lab (DDL) at Stanford University and with Prof. Jason Millar and Sophie Le Page from the Faculty of Engineering at the University of Ottawa.
Research results and further challenges
In this project, the investigation of value-based decision-making has highlighted some fundamental challenges in the development of autonomous road vehicles. The original issues can be readily illustrated in a thought experiment: Imagine you are driving next to a row of longitudinally parked vehicles in an urban environment. Each vehicle conceals a space that you cannot see. If a child then happens to run out onto the road in front of you at the worst possible moment, you will not be able to brake in time.
A more detailed analysis shows that decisions in road traffic are influenced by values: Do I want to reach an appointment as quickly as possible, or do I want to drive as safely as possible? My behavior changes depending on which value (safety versus speed) is at the focus of my attention. Discussion of this scenario of driving past parked vehicles vividly illustrates the inherent risk we expose ourselves to when we operate a vehicle ourselves and thus participate in road traffic. If we drive in accordance with standard practice, in very rare scenarios we will not be able to brake in time.
Autonomous road vehicles are likewise exposed to this risk, which is inherent to the traffic system. The perception systems of these new road users, just as for human drivers in the described situation, also have to contend with concealed spaces and the associated uncertainties: What road users are there in these spaces? What are their intentions? And what will they do next? Two different inherent risks that we have identified for autonomous road vehicles are already manifest in this scenario: the risk resulting from imperfect perception (concealment) and the risk resulting from uncertainty regarding the future actions of other possible road users.
In addition, from a technical perspective our transportation system is an open world, since almost anything can happen at any time. In view of its complexity, this open world is never fully describable for the developers of autonomous vehicles: Unforeseen effects between sensors and materials, for example, can mean that a sensor cannot “see” the fabric of an article of clothing due to its unfavorable reflective properties. Road users can wear highly creative costumes for Carnival celebrations, with the result that the cameras cannot recognize them as pedestrians. Even “new” road users, such as the now widespread e-scooters, can make an appearance in traffic after the launch of a new system, so that the machine perception systems fail to recognize them if they are not accounted for in the training data. These are just a few examples that demonstrate that even with the most conscientious and thorough development processes for autonomous vehicles, there will always be scenarios that are neither described nor tested during development. The incomplete nature of specifications and tests is a risk inherent to the system, resulting from the fact that a technical system must come to terms with an open world without human supervision. Formulation of these inherent risks is an important aim of this project.
The risks described above have now been kept to a minimum by means of prudent development and the use of systems engineering methods. However, due to their very nature they cannot be eliminated completely; residual risks remain. The studies in the “Value-based decision-making” project indicate that the methods used from the field of systems engineering are necessary but not sufficient to engender acceptance of autonomous road vehicles among the general public. They should be supplemented by methods of value-based development, as described in the graphic:
This graphic models the principal components of value-oriented development. The box at top right symbolizes the decision-making in the autonomous vehicle, which controls it in accordance with explicitly formulated values. In our specific example described at the outset, methods from so-called model predictive control make it possible to accommodate boundary conditions and thus to take the identified values into account: The blue triangle, for example, symbolizes the concealed area ahead of the parked car. The red field next to it represents a safety area introduced by the model predictive control in order to reduce the risk for children who could jump out from the concealed area in the described scenario. The planned travel path proceeds directly adjacent to this safety area.
How does the vehicle management know what values are important in a specific scenario from the perspective of those involved? Relevant stakeholders in this scenario are for example the vehicle occupants, all other road users, their relatives, the vehicle manufacturer, representatives of the state, and insurance companies. The values relevant to these parties must be identified by the company’s researchers and developers during the development phase (box at top left). The development paradigm of “value-sensitive design” appears particularly suitable for identifying these values. The systematic transfer of the values from the development process (box at left) to the area of decision-making (box at right) is taken into account in the formalization of requirements (box at top center); the latter has only been provisionally outlined to date and requires further research.
Since autonomous vehicles are to prove themselves in future on public roads, we regard clarification of values, and specifically resolution of conflicts in the basic configuration (in this example: speed versus safety), as being so essential that they should be negotiated together with representatives from society. This process of negotiation is depicted by the double arrow between the boxes at top left and bottom center. Relevant stakeholders are involved here in the early stages of development. In particular, questions regarding residual risks must be discussed carefully and in detail with representatives of the public. We are thus following a recommendation of the business ethicist Karl Homann from 2002: In his lecture on “Business and social acceptance: Driver assistance systems on the test bench” – at that time with reference to systems incorporating an automatic emergency brake – he already called for a transparent discussion of all operational risks with appropriate stakeholders in society prior to the market launch of corresponding series-produced systems.
