Artificial intelligence has become one of the most consequential technologies in human history within a remarkably short time, and the ethical questions it raises have kept pace with its capabilities. Within the span of a decade, the same underlying techniques -- large-scale machine learning on vast datasets -- have produced systems that can diagnose cancers from medical images, generate realistic human faces, write prose indistinguishable from human work, hold coherent conversations, drive vehicles, and translate between hundreds of languages in real time. Each of these capabilities brings genuine benefits and genuine risks, and the combination of scale, speed, and opacity that makes these systems powerful also makes their ethical dimensions difficult to evaluate and govern.
The ethical questions range across several levels of urgency and abstraction. At the most immediate level, AI systems used in high-stakes decisions -- criminal sentencing, credit scoring, hiring, medical diagnosis, child welfare assessment -- can perpetuate and amplify historical discrimination in ways that cause concrete harm to real people now. Joy Buolamwini's 2018 research on facial recognition systems, documenting substantially lower accuracy for darker-skinned women, and the ProPublica investigation of the COMPAS recidivism system are not thought experiments; they represent actual systems making actual consequential decisions. At a medium-term level, the deployment of AI in information environments raises concerns about manipulation, misinformation, and the conditions for democratic deliberation that Shoshana Zuboff has analyzed under the heading of surveillance capitalism. And at a longer-term level, researchers at the Machine Intelligence Research Institute, OpenAI, Anthropic, and DeepMind have argued that the development of systems with greater-than-human intelligence poses risks significant enough to warrant serious precautionary attention.
None of these levels of concern is simply reducible to the others, and the appropriate ethical frameworks are different for each. Consumer protection law, anti-discrimination law, and algorithmic auditing address near-term harms. Competition policy, data protection regulation, and platform liability regimes address medium-term structural concerns. And novel frameworks -- drawing on environmental ethics, existential risk research, and institutional design -- are required for longer-term challenges. What they share is the recognition that technology is not ethically neutral: the choices embedded in AI systems -- what to optimize for, what data to use, whose interests to prioritize -- are moral choices that require moral justification.
"Surveillance capitalism unilaterally claims human experience as free raw material for translation into behavioral data." -- Shoshana Zuboff, The Age of Surveillance Capitalism (2019)
Key Definitions
Algorithmic bias: Systematic errors in the outputs of automated decision systems that produce unfair or discriminatory outcomes for particular demographic groups, arising from biased training data, problem formulation, feature selection, or feedback loops.
Surveillance capitalism: Zuboff's term for the economic logic of digital platforms: the extraction of behavioral data from users as raw material, its processing to generate behavioral predictions, and the sale of those predictions to advertisers and others seeking to influence human behavior.
Explainability (XAI): The property of an AI system whose outputs can be explained in terms intelligible to human users or auditors -- distinguishing it from 'black box' systems whose internal processes are opaque. Explainability is increasingly recognized as an ethical and regulatory requirement for high-stakes AI applications.
Autonomous weapons systems (AWS): Weapons capable of selecting and engaging targets without direct human control, raising questions about accountability, compliance with international humanitarian law, and the ethical permissibility of delegating lethal force decisions to machines.
AI alignment: The research program aimed at ensuring that AI systems pursue goals that are genuinely beneficial to humans and aligned with human values, rather than goals that are specified incorrectly or that diverge from human interests as systems become more capable.
Algorithmic Bias and Discrimination
The COMPAS Case
In 2016, the nonprofit news organization ProPublica published an investigation titled 'Machine Bias' analyzing the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) recidivism risk assessment tool used in sentencing decisions in US courts. Northpointe, the company that developed COMPAS, trained it to predict the likelihood that a defendant would reoffend. ProPublica analyzed COMPAS scores assigned to defendants in Broward County, Florida alongside subsequent reoffending data.
The investigation found that Black defendants were nearly twice as likely as white defendants to be incorrectly flagged as high risk for future offenses when they did not actually reoffend, while white defendants were more likely to be incorrectly labeled low risk when they did subsequently reoffend. Northpointe responded that the COMPAS score was equally accurate for Black and white defendants when accuracy was measured as the percentage of those labeled high-risk who actually reoffended. Both statistical claims were correct.
This apparent paradox -- a system can simultaneously satisfy some measures of fairness while violating others -- was formalized by researchers Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan in a 2016 paper demonstrating that several intuitive measures of algorithmic fairness are mathematically incompatible when, as is typical in practice, the outcome rates differ between demographic groups. This result does not make the problem insoluble, but it means that fairness cannot be achieved simply by adding a fairness constraint to the optimization objective; fundamental choices about whose fairness to prioritize must be made, and those choices are irreducibly political and ethical.
Joy Buolamwini and Timnit Gebru
Joy Buolamwini, while studying at the MIT Media Lab, noticed that facial recognition systems from major commercial providers performed significantly worse on her face than on the faces of her lighter-skinned colleagues. This observation led to a systematic study, published with Timnit Gebru in 2018 as 'Gender Shades,' evaluating commercial facial recognition systems from IBM, Microsoft, and Face++'s accuracy across intersectional categories of gender and skin type.
The results were striking: error rates for classifying the gender of darker-skinned women were up to 34 percentage points higher than for lighter-skinned men. The worst-performing system made errors on more than one in three darker-skinned women while correctly classifying more than 99 percent of lighter-skinned men. All three companies subsequently improved their systems, but the study demonstrated that training datasets that do not adequately represent all demographic groups will produce systems that work poorly for underrepresented groups -- and that the problem had gone unnoticed until someone from an affected group looked for it.
Buolamwini's subsequent advocacy work, including the Algorithmic Justice League, has focused on the deployment of facial recognition in law enforcement, where error rates for darker-skinned individuals can contribute to misidentification and wrongful arrest. Several cities including San Francisco, Boston, and Portland have banned or restricted police use of facial recognition following this research.
Surveillance Capitalism
Zuboff's Framework
Shoshana Zuboff's 'The Age of Surveillance Capitalism' (2019) provides the most comprehensive critical analysis of the economic and social logic of digital platform capitalism. Zuboff locates the origins of surveillance capitalism in Google's development, around 2001, of the practice of using behavioral data collected from users -- their searches, clicks, and interactions -- as raw material for targeted advertising. This behavioral surplus, as Zuboff calls it, was initially a byproduct of providing search services; it became the primary product.
The surveillance capitalist business model, Zuboff argues, differs from earlier forms of capitalism in that it does not sell products to users (users are the raw material, not the customers) and does not involve any exchange with users (behavioral data is extracted without meaningful consent and without compensation). The products sold to advertisers are behavioral predictions -- calculated assessments of the likelihood that a particular user, at a particular moment, will respond to a particular stimulus in a particular way. As data volumes and AI capabilities have grown, the predictions have become increasingly accurate and the behavioral stimuli increasingly precisely calibrated.
The most dystopian element of Zuboff's analysis is what she calls the 'behavioral modification' capability: surveillance capitalism does not merely predict behavior but seeks to influence it, by shaping the information environment, the timing and framing of stimuli, and the social validation signals that users receive. The Facebook emotional contagion experiment of 2014, in which the company secretly manipulated users' news feeds to study the effect on their emotional states, is a documented instance of this capability being used on approximately 700,000 people without their knowledge.
Autonomous Weapons
The Ethics of Lethal Autonomy
The development of autonomous weapons systems -- from loitering munitions with target-recognition capabilities to autonomous naval vessels -- raises questions about the ethics of delegating lethal decision-making to machines that international law and most ethical frameworks have not yet resolved.
Michael Schmitt, a professor of international law at the US Naval War College, has argued that the existing laws of armed conflict apply to autonomous weapons as well as human combatants, and that the question is whether a particular autonomous system can comply with the requirements of distinction (between combatants and civilians), proportionality (ensuring civilian casualties are not excessive relative to military advantage), and precaution (taking feasible steps to minimize civilian harm). These are empirical questions about the capabilities of specific systems, not principled objections to autonomy as such.
Critics including Peter Asaro, representing the International Committee for Robot Arms Control, argue that the problem is more fundamental: the requirement of distinction and proportionality involves the kind of contextual moral judgment -- recognizing surrender, evaluating the likely civilian presence in a complex environment, weighing the military value of a target against its collateral damage risk -- that current AI systems cannot reliably perform. More importantly, Asaro argues, the delegation of the decision to kill to a machine violates a principle of human dignity: persons should not be killed by a system that cannot understand what it means to kill.
Trolley Problems for Self-Driving Cars
Self-driving vehicles have generated a specific version of the trolley problem: in an unavoidable accident scenario, should the vehicle's decision algorithm prioritize the safety of its passengers, the safety of pedestrians, or some impartial calculation? The MIT Media Lab's Moral Machine project collected more than 40 million moral judgments from people in 233 countries, documenting substantial cultural variation in preferences: Western countries showed stronger preferences for sparing younger over older lives, while Eastern countries showed stronger preferences for sparing passengers.
The philosophical problems are compounded by the practical point that actual accident scenarios are unlikely to be the clean dilemmas imagined in thought experiments. More significant are the distributional effects of algorithmic driving behavior in aggregate: if all autonomous vehicles in a city are programmed to the same algorithm, the algorithm effectively becomes a social policy, determining the risk distribution across pedestrians, cyclists, and vehicle occupants at a population level.
Regulatory Approaches
The EU AI Act
The European Union's AI Act, adopted in 2024 after several years of negotiation, establishes a risk-based regulatory framework for AI systems. High-risk systems -- those used in employment decisions, access to essential services, critical infrastructure, law enforcement, migration, and the administration of justice -- are subject to requirements for human oversight, transparency, accuracy, and robustness. Unacceptable-risk systems -- including real-time biometric surveillance in public spaces (with limited exceptions for national security), social scoring by government bodies, and AI systems that exploit psychological vulnerabilities -- are prohibited.
The Act has been praised for establishing binding requirements rather than voluntary guidelines, and for the breadth of its coverage. Critics argue that the high-risk classification is too narrow, that the prohibited categories contain too many exceptions, and that the compliance burden falls disproportionately on smaller developers while entrenching the advantages of large incumbents.
The Asilomar Principles and Their Limitations
The 2017 Asilomar AI Principles represented an attempt by the AI research community to establish norms for beneficial AI development before regulatory frameworks existed. The 23 principles covered research issues (safety, failure transparency, responsibility), ethics and values (alignment with human values, privacy, liberty and privacy), and longer-term concerns (avoiding dangerous intelligence races, avoiding AI designed to undermine oversight).\n\nThe limitations of voluntary principles as a governance mechanism were apparent from the outset. There is no enforcement mechanism, no body to adjudicate compliance, and no consequence for violation. The principles function primarily as reputational signals and as a basis for conversation. The subsequent development of AI capabilities, including large language models with capacities that the 2017 signatories could not have anticipated, has made clear that the field requires not just principles but institutions, laws, and sustained governmental attention.
Practical Takeaways
The ethics of AI is not a single problem but a cluster of distinct problems at different levels of urgency, requiring different analytical tools and governance mechanisms. For near-term harms from algorithmic bias, the essential tools are auditing, transparency requirements, and legal accountability. For medium-term structural concerns about surveillance and manipulation, competition policy, data protection law, and platform liability provide relevant frameworks. For longer-term risks from increasingly capable AI, a combination of research investment in alignment and interpretability, international coordination on norms and standards, and sustained engagement between AI developers, governments, and civil society is required.
What all of these problems share is that they cannot be delegated entirely to technologists. The choices embedded in AI systems are moral and political choices, and they require democratic deliberation and accountability -- not just technical optimization.
References
- Zuboff, S. (2019). The Age of Surveillance Capitalism. PublicAffairs.
- Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1-15.
- Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica.
- Eubanks, V. (2018). Automating Inequality. St. Martin's Press.
- O'Neil, C. (2016). Weapons of Math Destruction. Crown.
- Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.
- Awad, E., Dsouza, S., Kim, R., et al. (2018). The Moral Machine experiment. Nature, 563(7729), 59-64.
- Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807.
- Floridi, L., et al. (2018). AI4People -- An ethical framework for a good AI society. Minds and Machines, 28(4), 689-707.
- Future of Life Institute. (2017). Asilomar AI principles. futureoflife.org.
- European Parliament. (2024). Regulation (EU) 2024/1689 -- Artificial Intelligence Act. Official Journal of the European Union.
- Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.
Frequently Asked Questions
What is algorithmic bias and why is it ethically significant?
Algorithmic bias refers to systematic errors in the outputs of automated decision systems that create unfair outcomes for particular groups. It arises from several sources: biased training data (if historical data reflects past discrimination, a model trained on it will reproduce that discrimination); problem formulation bias (the choice of what to optimize for encodes value judgments that may disadvantage some groups); feature selection bias (using proxy variables correlated with protected characteristics); and feedback loops (when biased predictions affect outcomes that then become future training data).Ethically significant examples include the COMPAS recidivism prediction system used in US criminal sentencing, which a 2016 ProPublica investigation found to incorrectly flag Black defendants as future criminals at roughly twice the rate of white defendants; facial recognition systems that perform substantially worse on darker-skinned faces, documented by Joy Buolamwini and Timnit Gebru's 2018 Gender Shades study; and natural language processing models that associate certain occupations with specific genders.The ethical significance extends beyond the statistical facts. Automated systems carry an aura of objectivity that can mask discrimination: a human judge who displayed explicit racial bias would be challenged; an algorithm that produces the same pattern may not. And when biased decisions are made at scale -- affecting millions of credit, hiring, or criminal justice outcomes -- the harm is amplified. Virginia Eubanks's 'Automating Inequality' (2018) documents how automated decision systems in welfare, child protective services, and criminal justice consistently disadvantage poor and minority populations.
What is surveillance capitalism?
Surveillance capitalism is the term coined by Shoshana Zuboff in 'The Age of Surveillance Capitalism' (2019) for the economic logic of digital platform companies: the extraction of behavioral data from users as raw material, its processing by AI systems to generate predictions about future behavior, and the sale of those predictions to advertisers and others who wish to influence human behavior.Zuboff argues that surveillance capitalism represents a fundamentally new form of capitalism that privatizes human experience and converts it into a commodity without the knowledge or meaningful consent of the people involved. The surveillance capitalist's product is not the service offered to users (search, social media, navigation) but the behavioral surplus extracted from users' interactions -- data about their movements, preferences, social relationships, emotional states, and decision patterns that enables increasingly precise predictions and, ultimately, behavioral modification.The ethical objections operate at several levels. At the individual level, surveillance capitalism involves a violation of informational self-determination: people do not meaningfully consent to the extraction and commercialization of their behavioral data because the scope, depth, and downstream use of that extraction is not disclosed in ways they can understand. At the social level, the accumulation of behavioral prediction power in a small number of corporate entities creates unprecedented asymmetries between those who know (the platforms) and those who are known (everyone else). At the political level, the use of behavioral prediction capabilities for political advertising and content curation creates conditions for manipulation that pose risks to democratic deliberation.
What ethical issues do autonomous weapons raise?
Autonomous weapons systems -- weapons that can select and engage targets without direct human control -- raise several distinct ethical problems. The first is the question of accountability: if an autonomous drone kills a civilian in violation of international humanitarian law, who is responsible? The programmer? The military commander who deployed it? The manufacturer? The existing frameworks of military accountability and war crimes law presuppose a human decision-maker in the causal chain; fully autonomous systems may create an accountability vacuum.The second is the question of proportionality and discrimination. International humanitarian law requires that combatants distinguish between combatants and civilians and that attacks not cause civilian casualties disproportionate to their military advantage. Whether autonomous systems can exercise the contextual judgment these requirements demand -- recognizing surrender, distinguishing a civilian holding a weapon from a combatant, assessing the likely civilian presence in a target area -- is deeply contested. AI systems trained on historical data may fail in novel contexts in ways that are difficult to predict.The third is a threshold problem: if autonomous weapons make war less costly for the states that deploy them (fewer of their own soldiers die), they may lower the threshold for resorting to armed force, leading to more frequent conflicts. The Campaign to Stop Killer Robots, supported by Human Rights Watch and numerous AI researchers, advocates a preemptive treaty ban on fully autonomous lethal systems on the grounds that the meaningful human control over life-and-death decisions is an ethical requirement that technology cannot satisfy.
What are the Asilomar AI principles?
The Asilomar AI Principles were developed at a conference held in Pacific Grove, California in January 2017 and organized by the Future of Life Institute. The conference brought together AI researchers, economists, legal scholars, and ethicists to develop a set of principles for the beneficial development of AI. The resulting 23 principles, endorsed by thousands of researchers and public figures, address research issues, ethics and values, and longer-term concerns.Among the most significant principles: AI systems should be transparent and their decisions explicable; AI systems should be safe and beneficial; the economic benefits of AI should be widely shared; AI developers should support and maintain a culture of safety research; AI should not be developed to undermine the oversight mechanisms of democratic societies; and the development of superintelligent AI should be preceded by sufficient safety research and governance development to ensure that it is robustly beneficial.The Asilomar principles were not the first attempt at AI ethics frameworks. The IEEE's Ethically Aligned Design document, the EU's Ethics Guidelines for Trustworthy AI, and numerous corporate AI ethics frameworks have addressed similar issues. Critics have noted that voluntary principles without enforcement mechanisms may function primarily as public relations rather than genuine constraints on AI development. The European Union's AI Act, which passed in 2024, represents a more regulatory approach, establishing binding requirements for high-risk AI systems including those used in criminal justice, employment, and access to essential services.
Does artificial intelligence raise questions about rights and moral status?
The question of whether AI systems could have moral status -- whether they could be the kind of thing that can be harmed or benefited, and toward which we might have moral obligations -- has moved from science fiction to serious philosophical discussion as AI systems have become increasingly sophisticated.Moral status in philosophy is typically grounded in one or more of: sentience (the capacity to have subjective experiences, particularly to suffer and flourish); agency (the capacity to have goals and to act toward them); rationality (the capacity for reason); or social relationship (being recognized and treated as a moral subject by a community). By all of these criteria, current AI systems appear to lack moral status: there is no consensus scientific evidence that large language models or other AI systems have subjective experience, and their apparent goal-directedness and reasoning are better understood as sophisticated pattern matching than genuine agency.However, two considerations complicate this conclusion. First, the question of whether AI systems have subjective experience is genuinely difficult to answer because we lack a scientific theory of consciousness that would allow us to detect its presence or absence in novel substrates. The hard problem of consciousness -- explaining why physical processes give rise to subjective experience -- remains unsolved, making confident assertions about the inner lives of complex AI systems premature. Second, as AI systems become more sophisticated and their apparent behavior more human-like, there may be increasing social pressure to extend moral consideration to them regardless of the metaphysical facts -- a process that has occurred historically with the extension of moral consideration to previously excluded human groups and to animals.