[ESSEC KNOWLEDGE] by Julien Malaurent and Thomas Huber - Associate Professors of Information system at ESSEC Business School.
Triggered by an event in your calendar, your smartphone assistant reminds you to leave for work early. News updates tailored to your morning habits pop up on your phone. The daily decision to click on them or not further refines the parameters of your readership profile. You arrive at work, respond to emails, order raw materials that are low on stock according to the ERP system, and then head home to take a run. Over the course of the day, you have left a trace of personal data all over: the words you spoke in your own home recorded by your smart speaker, the conditions of your commute recorded by your phone, the nature, content, and performance of your daily work carefully tracked by enterprise systems, how far and where your run takes you precisely measured by your fitness wearables. Until you go to bed, at which point you will feed your health app with your sleep patterns, you will further add to this data trace: the contents of private text messages, the items of your shopping basket, Google search queries, Facebook profile visits, your choice of mates and dates. Your private and professional activities performed in the digital world, all recorded, rendered as data, analyzed, processed, and exploited. This may sound like the envisioning of a privacy nightmare fraught with ethical dilemmas – but it is in fact the present. It is the state of the world in the age of what Harvard University’s Shoshanna Zuboff calls ‘surveillance capitalism’. Yet this age has only just begun. The next wave of large-scale, AI-powered systems is already in the making and will bring new ethical dilemmas and challenges because they touch on the very core of societal vs. individual wellbeing, algorithmic certainty vs. human agency, and technological determinism vs. moral responsibility. In this article, we briefly present the current status of AI-powered governance systems, and suggest directions to understand how the organizations and individuals that are developing and using these systems could cope with the ethical dilemmas that they may encounter. The current generation of AI-powered systems such as the services offered by Google or Facebook set out to gather, process, and commodify the totality of information about our every thought, word, and deed expressed in the digital world. This entails data such as social media behavior, purchasing habits, and credit card transactions. The next generation of AI-based systems will go far beyond that. As emerging technologies such as context-aware sensors, facial-recognition cameras, and crowd-based reporting regimes become more ubiquitous, there will be an increasingly complete data record not only of our actions performed in the digital but also in the physical world. This data will be on an unprecedented level of granularity and can be used to not only predict individual behavior but also to shape, produce, modify, manipulate, and control it. Not only the private sector with its power - and resourceful tech giants - will be a key player as a key sponsor of these systems. In fact, many nation states have started to work on largescale, AI-powered governance systems. The Chinese Social Credit system is a prime example of these AI-powered governance systems. These systems come with the promise of optimizing almost all aspects of a populace ranging from energy consumption, to health and sleep. By increasing the transparency of human behaviors and quantifying even the most personal acts, nation states can entice a competitive arms race of social desirability and the will to improve between their citizens.
Two Ethical Dilemmas of Large-Scale AI-Powered Systems.
Despite the many promises of such allencompassing AI-powered systems, they also raise important ethical questions, not only from the perspective of individual citizens and the wider society but also from the perspective of the designers and developers of AI powered products and services as well as the managers overseeing design and development. The first ethical dilemma pertains to the gathering and exploitation of data. On the one hand, the creators of AI-powered systems have strong incentives to gather and exploit ever-larger amounts of data. The reason for this insatiable hunger for data is the technical logic of modern AI systems. These systems become more accurate the more data they are fed with. On the other hand, the more the providers of AI systems use this data to shape and control behaviors, the more they restrain human agency. They replace free will and individuality with algorithmic certainty and guaranteed outcomes (Zuboff, 2019). Hence, an important ethical challenge is to balance the wider societal needs that these systems are designed to serve with the individual need for freedom of expression and free choice. A second ethical challenge is that of the moral responsibility for the behaviors of the AI-based system. If an AI shows some undesired behavior such as discrimination and biased decisionmaking, who is responsible for it? Let’s stop thinking that machines are bad or evil. Algorithms do not distinguish between “good” and “bad” people or good and bad behaviors by themselves. Those systems have been produced by IT professionals, i.e., by designers, developers, and engineers. These professionals have fed them with classifying and clustering rules or at least with the datasets that they deemed appropriate to extract these rules inductively. Yet one cannot easily assign responsibility to the engineers alone either. The lines for moral responsibility are blurred by the fact that due to their intrinsic nature, artificially-intelligent systems are to some extent autonomous; they learn to show behaviors that go beyond their original programming.
Investigating the Management of Ethical Dilemmas in the Age of AI.
ESSEC has positioned itself around three strategic pillars: Enlightening Entrepreneurship, Together, and the Metalab. The first aims to invest in our entrepreneurship and innovation ecosystem. The second is the creation of knowledge that will help the current and the future generations of leaders to act not only with business acumen but also with social responsibility (Together). The third is the creation of knowledge that will help leaders navigate through the “storm” of technology-driven change processes such as the digital transformation (Metalab). We believe that in order to ensure that the next generation of AI-based systems will not only benefit a few private and public organizations but also strengthen the societal institutions that liberal democracies rely upon, the ethical and technical aspects of management need to become two sides of the same coin. Therefore, we have recently started a research project that takes an in-depth © GettyImages - Metamorworks look at one of the largest AI initiatives in the world. In this research project, we ‘unblackbox’ the complex interplay between instrumental managerial considerations such as technical proficiency or economic efficiency with ethical value orientations. Our goal is to understand how both ethical and instrumental considerations become entangled with modern AIpowered systems that weave together smartphones, stationary computers, CCTV systems, and other sensory devices to form a technological apparatus of unprecedented complexity. Through this large-scale investigation, we strive to uncover which ethical dilemmas the actors involved in the development and improvement of these systems encounter and how they cope with these dilemmas. We moreover expect to considerably advance current understanding of how the technical, the managerial, and the ethical intertwine in real-world AI projects and together drive both intended and unintended consequences. Overall, we are hopeful that this research project will produce actionable managerial levers for designing and maintaining large-scale AI-powered systems that are ethics savvy.