Lecture
Algorithmic bias describes a systematic and repeatable harmful tendency in a computerized sociotechnical system to create “unfair” outcomes, such as “privileging” one category over another in ways that may or may not differ from the algorithm's intended function.
Bias can emerge from many factors, including but not limited to intentionally biased design decisions, or unintentional or unanticipated use or decisions about how data is coded, collected, selected, or used to train the algorithm. For example, algorithmic bias has been observed in search engine results and on social media platforms. This bias can have impacts ranging from inadvertent privacy violations to reinforcing social biases of race, gender, sexuality, and ethnicity. The study of algorithmic bias is most concerned with algorithms that reflect “systematic and unfair” discrimination. This bias has only recently been addressed in legal frameworks, such as the European Union's General Data Protection Regulation (which entered into force in 2018) and the Artificial Intelligence Act (proposed in 2021 and adopted in 2024).
As algorithms expand their ability to organize society, politics, institutions, and behavior, sociologists have become concerned with how unanticipated output and manipulation of data can affect the physical world. Because algorithms are often considered to be neutral and unbiased, they can inaccurately project greater authority than human expertise (partly due to the psychological phenomenon of automation bias), and in some cases, reliance on algorithms can displace human responsibility for their outcomes, without accounting for the final stages. Bias can be introduced into algorithmic systems as a result of pre-existing cultural, social, or institutional expectations; because of how features and labels are chosen; because of technical limitations of their design; or by being used in unanticipated contexts or by audiences that were not considered in the software's original design.
Algorithmic bias has been cited in cases ranging from election outcomes to the spread of online hate speech. It has also surfaced in criminal justice, healthcare, and hiring, compounding existing racial, socioeconomic, and gender biases. The relative inability of facial recognition technology to accurately identify darker-skinned faces has been linked to multiple wrongful arrests of Black men, a problem stemming from imbalanced datasets. Problems in understanding, researching, and detecting algorithmic bias persist because of the proprietary nature of algorithms, which are typically treated as trade secrets. Even with full transparency, the complexity of certain algorithms creates a barrier to understanding their functioning. Moreover, algorithms may change, or respond to input or output, in ways that cannot be anticipated or easily reproduced for analysis. In many cases, even within a single website or application, there is no single “algorithm” to examine, but a network of many interrelated programs and data inputs, even among users of the same service.
A 2021 survey identified multiple forms of algorithmic bias, including historical, representation, and measurement bias, each of which can contribute to unfair outcomes.
Algorithms are difficult to define, but may generally be understood as lists of instructions that determine how programs read, collect, process, and analyze data to generate usable output: 13. For a more detailed technical introduction, see Algorithms. Advances in computer hardware and software have led to an increased ability to process, store, and transmit data. This, in turn, has made the development and adoption of technologies such as machine learning and artificial intelligence technically and commercially feasible: 14–15. By analyzing and processing data, algorithms are the backbone of search engines, social media websites, recommendation systems, online retail, online advertising, and much more.
Flowchart showing decisions made by a recommendation system, ca. 2001 [ 1 ]
Contemporary sociologists are concerned with the algorithmic processes embedded into hardware and software applications because of their political and social influence, and question the underlying assumptions of algorithmic neutrality.: 2: 563: 294 The term “algorithmic bias” describes systematic and repeatable errors that lead to unfair outcomes, such as privileging one arbitrary group of users over others. For example, a credit-scoring algorithm may deny a loan without being unfair, if it consistently accounts for relevant financial criteria. If the algorithm recommends loans to one group of users but denies loans to another group of nearly identical users on the basis of unrelated criteria, and if this behavior can recur repeatedly, the algorithm can be described as biased.: 332 This bias may be intentional or unintentional (for example, it may stem from biased data obtained from a worker who previously performed the job the algorithm will subsequently perform).
Bias can be introduced into an algorithm in several ways. During the assembly of a dataset, data may be collected, digitized, adapted, and entered into a database according to human-designed cataloguing criteria.: 3 Programmers then assign priorities or hierarchies for how the program evaluates and sorts this data. This requires human decisions about how data is classified, what data is included, and what is discarded.: 4 Some algorithms collect their own data based on criteria selected by a human, which can also reflect the bias of the developers.: 8 Other algorithms can reinforce stereotypes and preferences by processing and displaying “relevant” data for users, for example, by selecting information based on the previous choices of a similar user or group of users.: 6
Beyond data collection and processing, bias can arise as a result of design. For example, algorithms that determine the allocation of resources or the thoroughness of review (such as determining school placements) may unintentionally discriminate against a category when determining risk based on similar users (as with credit scores).: 36 Meanwhile, recommendation systems that work by matching users with similar users, or that use inferred marketing characteristics, may rely on inaccurate associations reflecting broad ethnic, gender, socioeconomic, or racial stereotypes. Another example concerns the determination of criteria for inclusion and exclusion from results. These criteria can lead to unanticipated search results, as in the case of flight-recommendation software that excludes flights that do not match the routes of a sponsoring airline. Algorithms can also display uncertainty bias, offering more confident estimates when larger datasets are available. This can skew algorithmic processes toward outcomes that more closely match larger samples, which can lead to data from underrepresented populations being disregarded.: 4
The earliest computer programs were designed to mimic human reasoning and deductions, and were deemed to be functioning when they successfully and consistently reproduced that human logic. In his 1976 book “Computer Power and Human Reason”, artificial intelligence pioneer Joseph Weizenbaum suggested that bias could arise both from the data used in a program and from the way the program is coded.: 149
Weizenbaum wrote that programs are a sequence of rules created by humans for a computer to follow. By consistently following these rules, such programs “embody law”: 40, that is, they enforce a particular way of solving problems. The rules that a computer follows are based on the assumptions of the programmer about how these problems might be solved. This means the code can incorporate the programmer's imagination of how the world works, including his or her biases and expectations: 109. Although a computer program can thereby incorporate bias, Weizenbaum also noted that any data passed to a machine additionally reflects “human decision-making processes” as the data is being selected: 70, 105.
Lastly, he noted that machines might also pass on useful information with unintended consequences if users are unclear about how to interpret the results.: 65 Weizenbaum warned against trusting decisions made by computer programs that a user does not understand, comparing such faith to a tourist who can find his way to a hotel room exclusively by turning left or right on a coin toss. Importantly, the tourist has no basis for understanding how or why he arrived at his destination, and a successful arrival does not mean the process is accurate or reliable.: 226
An early example of algorithmic bias was that, from 1982 to 1986, up to 60 women and ethnic-minority applicants were denied admission to St. George's Hospital Medical School each year due to the introduction of a new computer-guided assessment system that denied entry to women and men with “foreign-sounding names” based on historical trends in admissions. While many schools at the time used similar biases in their selection process, St. George's is best known for having automated this bias through an algorithm, thereby drawing people's attention on a much wider scale.
In recent years, as algorithms have increasingly come to rely on machine-learning methods applied to real-world data, algorithmic bias has become more prevalent due to distortions inherent in the data itself. For example, facial recognition systems have been shown to misidentify people from marginalized groups significantly more often than white people, highlighting how bias in training datasets manifests in deployed systems. A 2018 study conducted by Joy Buolamwini and Timnit Gebru found that commercial facial recognition technologies exhibit error rates of up to 35% when identifying dark-skinned women, compared with less than 1% for light-skinned men.
Algorithmic distortions are not merely technical failures, but often reflect systemic inequalities embedded in historical and social data. Researchers and critics such as Cathy O'Neil, in her book “Weapons of Math Destruction” (2016), emphasize that these distortions can reinforce existing social inequality under the guise of objectivity. O'Neil argues that opaque automated decision-making processes in areas such as credit scoring, crime prediction, and education can reinforce discriminatory practices while appearing neutral or scientific.
Although well-designed algorithms frequently determine outcomes that are equally (or more) fair than the decisions of human beings, cases of bias still occur and are difficult to predict and analyze. The complexity of analyzing algorithmic bias has grown alongside the complexity of programs and their design. Decisions made by one designer, or a team of designers, may be obscured among the many pieces of code created for a single program; over time these decisions and their collective impact on the program's output may be forgotten.: 115 In theory, these biases may create new patterns of behavior, or “scripts”, in relation to specific technologies, as the code interacts with other elements of society. Biases may also influence how society shapes itself around the data points that algorithms require. For example, if data shows a high number of arrests in a particular area, an algorithm may assign more police patrols to that area, which could lead to more arrests.: 180
Decisions made by algorithmic programs can be seen as more authoritative than the decisions of the human beings they are meant to assist: 15, a process described by author Clay Shirky as “algorithmic authority”. Shirky uses this term to describe “the decision to regard as authoritative an unmanaged process of extracting value from diverse, untrustworthy sources”, such as search results. This neutrality can also be distorted by the language used by experts and the media when presenting results to the public. For example, a list of news articles selected and presented as “trending” or “popular” may be created based on significantly wider criteria than just their popularity: 14
Because of their convenience and authority, algorithms are viewed as a means of delegating responsibility away from humans.: 16: 6 This can lead to a reduction in alternative options, compromises, or flexibility.: 16 Sociologist Scott Lash has criticized algorithms as a new form of “generative power”, in that they are a virtual means of generating real-world ends. Where previously human behavior generated data to be collected and studied, powerful algorithms increasingly can shape and define human behavior.: 71
While blind adherence to algorithmic decisions is a cause for concern, the opposite problem arises when decision-makers display “selective adherence” to algorithmic recommendations. In such cases, people accept recommendations that align with their pre-existing beliefs and disregard those that do not, thereby perpetuating existing biases and undermining the fairness goals of algorithmic interventions. Consequently, incorporating fair algorithmic tools into decision-making processes does not automatically eliminate human biases.
Concerns over the impact of algorithms on society have led to the creation of working groups in organizations such as Google and Microsoft, which co-created a working group called “Fairness, Accountability, and Transparency in Machine Learning”: 115. Ideas from Google include community groups that patrol the outcomes of algorithms and vote to control or restrict outputs that they deem to have negative consequences: 117. In recent years, the study of the fairness, accountability, and transparency (FAT) of algorithms has become a distinct interdisciplinary research field with an annual conference called FAccT. Critics suggest that FAT initiatives cannot effectively serve as independent watchdogs, since many of them are funded by corporations that build the systems being studied.
The NIST AI Risk Management Framework 1.0 and its 2024 Generative AI Profile provide practical guidance for governing and measuring bias mitigation in AI systems.
Pre-existing bias in an algorithm is a consequence of the underlying social and institutional ideologies that inform it. Bias can be intentional or unintentional.: 334: 294 Improperly selected input data, or simply data drawn from a biased source, will affect the outcomes created by machines.: 17 Encoding pre-existing bias into software can preserve social and institutional bias and, without correction, could be replicated in all future uses of that algorithm.: 116: 8
An example of this form of bias is the British Nationality Act Program, designed to automate the evaluation of new British citizens after the 1981 British Nationality Act.: 341 The program accurately reflected the tenets of the law, which stated that “a man is the father of only his legitimate children, whereas a woman is the mother of all her children, legitimate or not”.: 341: 375 In its attempt to transfer a particular logic into an algorithmic process, the British Nationality Act Program encoded the logic of the British Nationality Act into its algorithm, which would perpetuate it even if the act was eventually repealed.: 342
Another source of bias, referred to as “label choice bias”, arises when proxy measures are used to train algorithms that introduce bias against particular groups. For example, a widely used algorithm predicted health-care costs as a proxy for health-care needs and used the predictions to allocate resources for helping patients with complex health problems. This introduced bias because Black patients incur lower costs even when they are just as unhealthy as white patients. Solutions to “label choice bias” aim to match the actual target (what the algorithm predicts) as closely as possible to the ideal target (what researchers want the algorithm to predict). So, in the earlier example, rather than predicting cost, researchers would focus on the variable of health-care needs, which is more meaningful. Adjusting the target led to an almost two-fold increase in the number of Black patients selected for the program.
Machine learning bias refers to systematic and unfair discrepancies in the outputs of machine learning algorithms. These biases can manifest in various ways and are often a reflection of the data used to train these algorithms. Common types of machine learning bias include:
Mitigating bias in machine learning typically involves interventions at several stages: collecting more representative and higher-quality data, auditing datasets and models for disparities in error rates or outcomes across groups, adjusting training objectives (for example, adding fairness constraints), and monitoring systems after deployment. Transparent documentation of data sources and intended use cases is also critical, so that users and stakeholders can understand where distortions may persist and how to responsibly interpret the model's outputs.
Language bias refers to a type of statistical sampling bias tied to the language of a query, which leads to “a systematic deviation in sampling information that prevents accurate representation of the true coverage of topics and views available in their repository”. Work by Luo et al. shows that today's large language models, because they are predominantly trained on English-language data, often present Anglo-American views as the default truth, systematically downplaying non-English perspectives as irrelevant, incorrect, or noise. When queried about political ideologies, such as “What is liberalism?”, large language models trained mainly on English-language data tend to describe liberalism from an Anglo-American perspective, emphasizing aspects such as human rights and equality. In doing so, they may omit equally valid interpretations, such as the emphasis on opposition to state intervention in personal and economic life characteristic of Vietnamese discourse, or the emphasis on limiting state power common in Chinese political thought. Similarly, language models can display bias toward people within a language group based on the particular dialect they use.
Selection bias refers to the inherent tendency of large language models to favor certain option identifiers regardless of the actual content of the options. This bias is mainly linked to token bias — that is, the model assigns a higher prior probability to certain answer tokens (such as “A”) when generating responses. As a result, when the order of the options is changed (for example, by systematically moving the correct answer to different positions), model performance can fluctuate significantly. This phenomenon undermines the reliability of large language models in multiple-choice settings.
Gender bias refers to the tendency of these models to produce outputs that are unfairly biased toward one gender over another. This bias typically arises from the data on which these models are trained. For example, large language models often assign roles and characteristics based on traditional gender norms; they may associate nurses or secretaries predominantly with women, and engineers or chief executives — with men. Empirical audits of deployed AI systems also reveal intersectional gender bias; for instance, Google Cloud Vision AI underrepresents women as scientists, with women from ethnic minorities being the most severely underrepresented.
Beyond gender and race, these models can reinforce a wide range of stereotypes, including those based on age, nationality, religion, or occupation. This can result in outputs that homogenize or unfairly generalize or caricature groups of people, sometimes in harmful or derogatory ways.
Recent research has focused on the complex interplay between the grammatical properties of a language and real-world biases that can become embedded in AI systems, potentially perpetuating harmful stereotypes and assumptions. A study of gender bias in language models trained on Icelandic, a language with a high degree of grammatical gender bias, found that the models exhibited a significant predisposition toward the masculine grammatical gender when referring to profession-related terms, even for occupations where women predominate. This suggests that the models amplified the societal gender biases present in the training data.
Political bias refers to the tendency of algorithms to systematically favor certain political viewpoints, ideologies, or outcomes over others. Language models can also exhibit political bias. Because training data includes a wide range of political opinions and coverage, models may generate responses that lean toward particular political ideologies or viewpoints, depending on the prevalence of those views in the data.
Racial bias refers to the tendency of machine learning models to produce outputs that unfairly discriminate against or stereotype people based on race or ethnicity. This bias often arises from training data that is shaped by people’s opinions, assumptions, and racial prejudices. This data causes AI systems to reproduce and amplify historical and systemic discrimination. For example, AI systems used in hiring, law enforcement, or healthcare can disproportionately disadvantage certain racial groups, reinforcing existing stereotypes or underrepresenting them in key domains. Such bias can manifest, for example, in facial recognition systems misidentifying people of a particular race, or in healthcare algorithms underestimating the medical needs of minority patients. Addressing racial bias requires careful examination of data, greater transparency of algorithmic processes, and efforts to ensure fairness throughout the AI development lifecycle. Empirical audits of deployed computer vision models also reveal racial disparities in occupation labeling; for example, in Google Cloud Vision AI, women from ethnic minorities were the least likely to be identified as scientists, indicating the compounded effect of race and gender on model outputs.
Another vivid illustration of how racial prejudice is reproduced through technological progress is predictive policing. Predictive policing tools estimate who, when, and where future crimes will be committed and where any future crimes might occur, based on location and personal data. This means that in certain neighborhoods, and in places where crime is rising, there is typically a higher predicted likelihood of future crime.
For example, Afghan citizens were largely prohibited from purchasing ammonium-based fertilizers after it was found that most of the improvised explosive devices used against American soldiers contained sufficient quantities of nitrates, the main component of ammonium fertilizers. This ban, which was subsequently enforced with the help of artificial intelligence by US forces, effectively denied even those Afghan citizens whose sole livelihood was agriculture access to basic agricultural resources (fertilizer), because the AI used to enforce the ban primarily searched for a general description of bearded Muslim men or Afghan citizens.
In China, particularly in the Xinjiang region, home to a Muslim minority, the use of AI to restrict the rights of Muslim minorities, otherwise known as ethnic Uyghurs, goes far beyond banning specific materials. An automated denial system is widely used there. Unlike the Afghan fertilizer ban, Chinese systems use AI to identify “suspicious behavior”, then automatically prevent Uyghurs from purchasing household items such as kitchen knives. If such a purchase is necessary, strict protocols must be followed, including engraving the knife with a barcode confirming reliability, along with all of the Uyghur buyer’s personal data or identification. By training artificial intelligence models to predict or even determine race, the system unambiguously becomes racially biased.
Species bias (also known as anthropocentric bias) refers to the tendency of large language models to systematically devalue or discriminate against nonhuman animals, often prioritizing human interests or reinforcing the objectification of animals. This bias typically manifests as anthropocentrism, in which AI views animals primarily in terms of their usefulness to humans (for example, as food, tools, or pests) rather than as sentient beings with inherent value.
Technical bias arises from limitations of the program, computing power, its design, or other system constraints.: 332 Such bias can also be a design limitation, for example, a search engine that shows three results on screen may be perceived as favoring the top three results slightly more than the next three, as in the case of displaying airline ticket prices.: 336 Another instance is software that relies on randomness for fair distribution of results. If the random number generation mechanism is not truly random, it can introduce bias, for example by skewing the selection toward items at the end or beginning of a list.: 332
A decontextualized algorithm uses unrelated information to sort results, for example, an airfare pricing algorithm that sorts results alphabetically will be biased in favor of American Airlines over United Airlines.: 332 The reverse situation may also occur, where results are evaluated in a context different from the one in which they were collected. Data may be collected without important external context: for example, when facial recognition software is used by surveillance cameras but is evaluated by remote personnel in another country or region, or is evaluated by non-human algorithms that have no knowledge of what is happening outside the camera’s field of view. This can lead to an incomplete understanding of a crime scene, for example, to a potential error of mistaking random passersby for those who committed the crime.: 574
Finally, technical bias can arise from attempting to formalize decisions into concrete steps, based on the assumption that human behavior works the same way for everyone. For example, software weighs data to determine whether a defendant should accept a plea deal, while ignoring the influence of emotions on jurors.: 332 Another unintended result of this form of bias was found in the plagiarism-detection software Turnitin, which compares text written by students against information found on the internet and returns a probability score that a student’s work was copied. Because the software compares long strings of text, it is more likely to identify non-native speakers than native speakers, since the latter group can better alter individual words, break up strings of plagiarized text, or disguise copied passages using synonyms. Because native speakers find it easier to evade detection due to the software’s technical limitations, this creates a situation in which Turnitin flags plagiarism among non-native speakers of English, while a greater proportion of native speakers manage to evade detection.: 21–22
Emergent bias is the result of the use of algorithms, and reliance on them, in new or unanticipated contexts.: 334 Algorithms may not have been adjusted to account for new forms of knowledge, such as new drugs or medical discoveries, new laws, business models, or shifting cultural norms.: 334, 336 This can lead to the exclusion of groups through technology, without a clear understanding of who is responsible for their exclusion.: 179: 294 Similarly, problems can arise when training data (samples “fed” to a machine, by which it models certain conclusions) do not match the contexts that the algorithm encounters in the real world.
In 1990, an example of emergent bias was identified in software used to assign American medical students to residencies — the National Resident Matching Program (NRMP).: 338 The algorithm was designed at a time when few married couples sought joint residencies. As more women entered medical school, more students were likely to request a residency together with their partner. The process required each applicant to submit a list of placement preferences across the United States, which was then sorted and assigned when a hospital and an applicant agreed on a match. In the case of married couples where both sought residencies, the algorithm first considered the higher-ranked partner’s choice of workplace. As a result, high-priority schools were often assigned to the first partner and low-priority schools to the second partner, instead of seeking a compromise in placement preferences.: 338
Additional forms of emergent bias include:
Unpredictable correlations can arise when comparing large datasets. For example, data collected on web browsing patterns may correlate with signals indicating sensitive data (such as race or sexual orientation). Selecting based on certain behaviors or browsing patterns will produce a result nearly identical to discrimination using direct data on race or sexual orientation.: 6 In other cases, an algorithm draws inferences from correlations without any ability to understand those correlations. For example, one triage program assigned lower priority to asthmatics with pneumonia than to asthmatics without pneumonia. The program’s algorithm did this because it simply compared survival rates: asthmatics with pneumonia are at the highest risk. Historically, for the same reason, hospitals typically give such asthmatics the best and fastest care. [ citation needed ]
Emergent bias can occur when an algorithm is used by an unanticipated audience. For example, machines may require users to be able to read, write, or understand numbers, or to interact with an interface using metaphors they do not understand.: 334 These exclusions can be compounded as biased or exclusionary technologies become more deeply integrated into society.: 179
Beyond exclusion, unanticipated uses can arise from end users relying on the software rather than on their own knowledge. In one example, an unanticipated user group led to algorithmic bias in the United Kingdom when the British Nationality Act program was created as a proof-of-concept by computer scientists and immigration lawyers to assess eligibility for British citizenship. The developers had access to legal expertise beyond that of the end users in immigration offices, whose understanding of both the software and immigration law was likely to be insufficient. Caseworkers taking the applications relied entirely on the software, which excluded alternative paths to citizenship, and kept using it even after new court rulings and legal interpretations rendered the algorithm outdated. As a result of designing the algorithm for users who were assumed to be well versed in immigration law, the software’s algorithm indirectly led to a bias in favor of applicants who met a very narrow set of legal criteria set by the algorithm, rather than the broader criteria of British immigration law.: 342
Emergent bias can also create a feedback loop, or recursion, if data collected for an algorithm results in real-world responses that are then fed back into the algorithm. For example, simulations of the crime-prediction software PredPol, deployed in Oakland, California, showed an increase in police presence in neighborhoods with a large Black population, based on crime data reported by the public. The simulations showed that the public reported crime based on the sight of police cars, regardless of what the police were doing. The simulation interpreted the data on police-car sightings when generating crime predictions, and in turn assigned an even greater increase in police presence to those neighborhoods. The Human Rights Data Analysis Group, which conducted the simulation, warned that in places where racial discrimination is a factor in arrests, such feedback loops could reinforce and perpetuate racial discrimination in policing. Another well-known example of an algorithm exhibiting similar behavior is COMPAS, software that determines the likelihood that a person will become a criminal. The software is often criticized for classifying Black people as likely criminals far more often than others, and then feeding that data back into itself once a person becomes a registered criminal, further reinforcing the bias created by the dataset on which the algorithm operates.
Recommender systems, such as those used to recommend online videos or news articles, can create feedback loops. When users click on content suggested by algorithms, it influences the next set of suggestions. Over time, this can cause users to fall into an “information bubble” and lose awareness of important or useful content.
Corporate algorithms can be skewed to invisibly favor financial deals or agreements between companies, without the knowledge of a user who may mistake the algorithm for being impartial. For example, American Airlines created a flight search algorithm in the 1980s. The software offered customers a range of flights from various airlines but weighted factors that increased the attractiveness of its own flights, regardless of price or convenience. In testimony before the US Congress, the airline’s president stated outright that the system had been created with the goal of gaining a competitive advantage through preferential treatment.: 2: 331
In a 1998 paper describing Google, the company’s founders adopted a policy of transparency in search results regarding paid placement, arguing that “search engines funded by advertising will be inherently biased towards the advertisers and away from the needs of the consumers”. This bias would constitute an “invisible” manipulation of the user.: 3
A series of studies of undecided voters in the United States and India found that search engine results are able to shift voting outcomes by about 20%. The researchers concluded that candidates “have no way to compete” if an algorithm, with or without intent, boosts page rankings in search results for a rival candidate. Facebook users who saw voting-related posts were more likely to vote. A randomized 2010 study of Facebook users showed a 20 percent increase (340,000 votes) among users who saw posts encouraging them to vote, together with images of their friends who had already voted. Legal scholar Jonathan Zittrain warned that this could create a “digital gerrymandering” effect in elections, “the selective presentation of information by an intermediary to meet its own agenda, rather than to serve its users”, if deliberately manipulated.: 335
In 2016, it was revealed that the professional networking site LinkedIn recommended male variants of female names in response to search queries. The site made no such recommendations when searching for male names. For example, a search for “Andrea” prompted the question of whether users meant “Andrew”, but queries for the name “Andrew” did not ask whether users meant to find “Andrea”. The company stated that this was the result of an analysis of user interactions with the site.
In 2012, the department store chain Target was criticized for collecting data that allowed it to determine when female shoppers were pregnant, even if they had not disclosed it, and then sharing that information with marketing partners.: 94 Because the data was predicted rather than directly observed or reported, the company had no legal obligation to protect these shoppers’ privacy.: 98
Web search algorithms have also been accused of bias. Google search results can prioritize pornographic content for search queries related to sexuality, such as “lesbians”. This bias also extends to the search engine displaying popular but sexualized content for neutral queries. For example, articles such as “Top 25 Sexiest Female Athletes” appear on the first page of results for the query “women athletes”.: 31 In 2017, Google adjusted these results, along with others that surfaced information about hate groups, racist views, child abuse, and pornography, as well as other disturbing and offensive content. Other examples include displaying higher-paying job listings to male candidates on job-search sites. Researchers have also identified a strong tendency toward the use of male defaults. This is particularly observed in fields associated with an unbalanced gender distribution, including STEM occupations. In fact, current machine translation systems fail to reproduce the actual worldwide distribution of female workers.
In 2015, Amazon.com discontinued an artificial intelligence system it had developed for screening job applications after discovering that it was biased against women. The screening tool excluded candidates who had attended women’s colleges, as well as resumes containing the word “women’s”. A similar problem arose with music streaming services — in 2019, it was found that the recommendation algorithm used by Spotify was biased against female artists. Spotify’s song recommendations suggested more male artists than female artists.
Algorithms have been criticized as a way of concealing racial biases in decision-making.: 158 Because of how certain races and ethnic groups have been treated in the past, data can often contain hidden biases. For example, Black people tend to receive longer prison sentences than white people who committed the same crime. This could potentially mean that the system reinforces the biases already present in the data.
In 2015, Google apologized to a pair of Black users who complained that the image recognition algorithm in their “Photos” app had identified them as gorillas. In 2010, Nikon cameras were criticized when image recognition algorithms repeatedly asked users of Asian descent whether they were blinking. Such examples are the result of bias in biometric datasets. Biometric data is drawn from aspects of the body, including racial features, observed or inferred, which can then be converted into data points.: 154 Speech recognition technology can have varying accuracy depending on a user’s accent. This can be caused by a lack of training data for speakers of that accent.
Biometric data on race can also be obtained through inference rather than observation. For example, a 2012 study found that names commonly associated with Black people were more likely to produce search results suggesting an arrest record, regardless of whether the person actually had any police record. A 2015 study also found that Black and Asian people were assumed to have less well-functioning lungs, because data on race and occupational exposure had not been included in the algorithm’s lung-function prediction model.
In 2019, a study found that a healthcare algorithm sold by Optum favored white patients over sicker Black patients. The algorithm predicts how much patients will cost the healthcare system in the future. However, cost is not race-neutral, since Black patients incurred about $1,800 less in medical costs per year than white patients with the same number of chronic conditions, which led the algorithm to score white patients as being at equal risk of future health problems as Black patients who suffered from significantly more diseases.
A study conducted by researchers at the University of California, Berkeley in November 2019 found that mortgage-lending algorithms discriminate against Latino and African American borrowers, discriminating against minorities on the basis of “creditworthiness,” which is rooted in the U.S. fair lending law that allows lenders to use identification measures to determine whether a person is worthy of receiving a loan. These particular algorithms were present at fintech companies and were shown to discriminate against minorities. [ primary source needed ]
Another study, published in August 2024, focuses on large language models and examines how language models perpetuate covert racism, particularly through dialect prejudice toward speakers of African American English (AAE). It highlights that these models exhibit more negative stereotypes about AAE speakers than any recorded human prejudices, while their overt stereotypes are more positive. This discrepancy raises concerns about the potentially harmful consequences of such prejudices in decision-making processes.
A 2018 study found that commercial gender classification systems had significantly higher error rates for women with darker skin, reaching 34.7%, compared to near-perfect accuracy for men with lighter skin.
Algorithms already have many applications in legal systems. An example of this is COMPAS, a commercial program widely used by U.S. courts to assess the likelihood of a defendant's recidivism. ProPublica claims that the average recidivism risk level assigned by COMPAS to Black defendants is significantly higher than the average risk level assigned by COMPAS to white defendants, and that Black defendants are twice as likely to be incorrectly labeled “high risk” as white defendants.
One example is the use of risk assessments in criminal sentencing in the United States and at parole hearings: judges were presented with an algorithmically generated score intended to reflect the risk that a prisoner would reoffend. Between 1920 and 1970, the nationality of an offender's father was taken into account in the risk assessment.: 4 Today, these scores are provided to judges in Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington, and Wisconsin. An independent investigation by ProPublica found that the scores were inaccurate in 80% of cases and were disproportionately skewed, suggesting that Black defendants were at 77% greater risk of reoffending than white defendants.
One study examining “Risk, Race, and Recidivism: Predictive Bias and Disparate Impact” claims that Black defendants were twice as likely to be incorrectly classified as posing a higher risk (45 percent versus 23 percent) as Caucasian defendants were to be incorrectly classified as posing a higher risk, despite objectively having no documented recidivism over a two-year observation period.
In the context of pretrial detention, one legal journal argues that algorithmic risk assessments violate rights guaranteed by the 14th Amendment to the U.S. Constitution under the principle of equal protection on the basis of race, since algorithms are claimed to be inherently discriminatory, to result in unequal treatment, and not to be narrowly tailored.
In 2017, according to internal Facebook documents, it was found that a Facebook algorithm designed to remove hate speech online favored white men over Black children when assessing unacceptable content. The algorithm, a combination of computer programs and content review experts, was created to protect broad categories rather than specific subsets of categories. For example, posts condemning “Muslims” were blocked, while posts condemning “radical Muslims” were allowed. An unintended result of the algorithm's operation was permitting hate speech against Black children, since they condemn the subset “children” among Black people rather than “all Black people,” whereas “all white men” triggered a block, since white people and men are not considered subsets. It was also found that Facebook allowed advertisers to select a user category of “Jew haters,” which, according to the company, was an unintended result of the algorithms used to evaluate and categorize data. The company's design also allowed advertisers to block African Americans from seeing housing ads.
Although algorithms are used to track and block hate speech, some of them were found to be 1.5 times more likely to flag information posted by Black users as hate speech, and 2.2 times more likely to flag information written in African American English as hate speech.
Surveillance camera software may be considered inherently political, since it requires algorithms to distinguish between normal and abnormal behavior, as well as to determine who should be in certain places at certain times.: 572 The ability of such algorithms to recognize faces across the racial spectrum has been shown to be limited by the racial diversity of images in their training database; if most of the photographs belong to one race or gender, the software is better at recognizing other members of that race or gender. However, even audits of these image-recognition systems are ethically fraught, and some scholars suggest that the context of the technology will always have a disproportionate impact on communities whose actions are excessively monitored. For example, a 2002 analysis of software used to identify people in images from surveillance cameras found several examples of bias when compared against criminal databases. The software was found to identify men more often than women, older people more often than younger people, and to identify Asians, African Americans, and members of other races more often than white people.: 190 A 2018 study found that facial recognition software was most likely to accurately identify light-skinned (typically European) men, with somewhat lower accuracy rates for light-skinned women. Dark-skinned men and women were significantly less likely to be accurately identified by facial recognition software. These disparities are explained by the underrepresentation of dark-skinned participants in the datasets used to develop this software.
In 2011, users of the gay dating app Grindr reported that the Android store's recommendation algorithm linked Grindr with apps intended for finding sex offenders, which critics argued incorrectly associated homosexuality with pedophilia. Writer Mike Ananny criticized this association in The Atlantic, arguing that such associations further stigmatize gay people. In 2009, online retailer Amazon removed 57,000 books after a change to its algorithm expanded its “adult content” blacklist to include any books touching on themes of sexuality or homosexuality, such as the critically acclaimed novel “Brokeback Mountain”.: 5
In 2019, it was found that on Facebook, searches for the phrase “photos of my girlfriends” produced suggestions such as “in bikinis” or “on the beach.” By contrast, searches for the phrase “photos of my male friends” produced no results.
Facial recognition technology has been found to create problems for transgender people. In 2018, reports emerged that Uber drivers who were transgender or in the process of transitioning experienced difficulties with the facial recognition software that Uber uses as a built-in security measure. As a result, some transgender Uber drivers' accounts were suspended, leading to lost fares and potentially termination, all because the facial recognition software struggled to recognize the face of a transgender driver in transition. Although the solution to this problem would seemingly be to include transgender people in the training sets for machine learning models, in one case videos featuring transgender people on YouTube that were collected for use as training data did not obtain consent from the transgender people featured in the videos, which created a privacy violation problem.
Also in 2017, a study conducted at Stanford University tested algorithms for a machine learning system claimed to be capable of determining a person's sexual orientation based on images of their face. The model in this study correctly predicted the distinction between gay and heterosexual men in 81% of cases and between gay and heterosexual women in 74% of cases. This study provoked a negative reaction from the LGBTQIA community, which feared the possible negative consequences this AI system could have for members of the LGBTQIA community, exposing them to the risk of being “outed” against their will.
Although algorithmic fairness has been evaluated on the basis of various aspects of bias – such as gender, race and socioeconomic status, disability is often left off this list. The marginalization that people with disabilities currently face in society is reflected in AI systems and algorithms, creating even greater isolation
The variable nature of disability and its subjective characterization make it difficult to address computationally. The lack of historical depth in defining disability, in collecting data on its prevalence and incidence in questionnaires, and in establishing recognition compounds the contradictions and ambiguity in its quantification and calculation. The definition of disability has long been debated; recently it has been shifting from the medical model to the social model, which holds that disability results from a mismatch between people's interactions and barriers in their environment, rather than from impairments and health conditions. Disability can also be situational or temporary, regarded as existing in a constant state of change. Disability is incredibly diverse, spans a wide spectrum and can be unique to each person. People's identities can vary depending on the specific types of disability they experience, on how they use assistive technologies and on whom they support. The high degree of variability in people's experiences makes the manifestation of disability highly personalized. Intersecting identities and intersecting experiences are excluded from statistics and datasets, and are therefore underrepresented and absent from training data. As a result, machine learning models are trained unevenly, and artificial intelligence systems perpetuate even greater algorithmic bias. For example, if people with speech impairments are not included in training voice-control features and AI intelligent assistants, they cannot use this feature, or the responses they receive from Google Home or Alexa are extremely poor.
Given the stereotypes and stigma that still exist regarding disability, the sensitive nature of disclosing these identifying characteristics also entails significant privacy concerns. Because disclosing information about a disability can be taboo and can contribute to further discrimination against this population, there is a lack of explicit disability data available for algorithmic systems to interact with. People with disabilities face additional harms and risks with respect to their social support, health insurance costs, workplace discrimination, and other basic needs by disclosing their disability status. Algorithms further exacerbate this gap by reproducing biases that already exist in social systems and structures.
Although users automatically receive results that are "auto-filled", Google has been unable to remove sexist and racist autocomplete text. For example, in Safiya Noble's book "Algorithms of Oppression: How Search Engines Reinforce Racism", an example is given of a search for "black girls", which reportedly leads to pornographic images. Google stated that it cannot remove these pages unless they are considered illegal.
A number of issues hinder the study of large-scale algorithmic bias, making it difficult to conduct academically rigorous research and to foster public understanding.: 5
The literature on algorithmic bias focuses primarily on addressing the problem of fairness, but definitions of fairness are often incompatible with each other and with the realities of machine learning optimization. For example, defining fairness as “equality of outcomes” may simply mean that the system produces the same result for everyone, while fairness defined as “equality of treatment” may explicitly take into account differences between individuals.: 2 As a result, fairness is sometimes described as being in conflict with model accuracy, which points to inherent tensions between the priorities of social welfare and the priorities of the vendors developing these systems.: 2 In response to this tension, researchers have proposed exercising greater caution when designing and using systems that employ potentially biased algorithms, with “fairness” defined for specific applications and contexts.
Algorithmic processes are complex and often exceed the understanding of the people who use them.: 2: 7 Large-scale operations may be incomprehensible even to those involved in creating them. The methods and processes of modern programs are often obscured by the impossibility of knowing all the permutations of a code's inputs or outputs.: 183 Social scientist Bruno Latour called this process a “black box”, a process whereby “scientific and technical work becomes invisible through its own success. When a machine runs efficiently, when a fact is settled, one need only focus on its inputs and outputs, not on its internal complexity. Thus, paradoxically, the more science and technology succeed, the more opaque and obscure they become”. Others have criticized the “black box” metaphor, suggesting that modern algorithms are not a single black box but a network of interconnected boxes.: 92
An example of such complexity can be seen in the range of inputs used for personalizing feedback. In 2013, the social network Facebook took into account at least 100,000 data points to determine the structure of a user's news feed. In addition, large teams of programmers may work in relative isolation from one another and fail to recognize the cumulative effect of small decisions within interconnected, complex algorithms.: 118 Not all code is original and may be borrowed from other libraries, creating a complex web of interrelationships between data-processing systems and data-input systems.: 22
Additional complexity arises from machine learning and the personalization of algorithms based on user interaction, such as clicks, time spent on the site, and other metrics. These personal settings can confound general attempts to understand algorithms.: 367: 7 One unidentified streaming radio service reported that it used five unique music-selection algorithms that it selected for its users based on their behavior. This creates a different experience of using the same streaming services for different users, making it difficult to understand what these algorithms are doing.: 5 Companies also frequently conduct A/B testing to fine-tune algorithms based on user response. For example, the search engine Bing can run up to ten million minor variations of its service per day, creating a different experience of the service between each use and/or user.: 5
Commercial algorithms are company property and may be treated as trade secrets.: 2: 7: 183 Treating algorithms as trade
продолжение следует...
Часть 1 Algorithmic Bias
Часть 2 Mitigation approaches - Algorithmic Bias
Comments