Algorithmic bias

In short

Algorithmic bias is when a computer system, including an AI model, produces results that are systematically unfair to some groups of people, for example by making more mistakes for women or for people with darker skin.

Also called: AI bias, Machine learning bias, Bias in AI
Topic: Safety, ethics and risk
Level: Beginner
First described: Batya Friedman and Helen Nissenbaum set out an early framework in "Bias in Computer Systems" (1996), sorting bias into pre-existing, technical and emergent types⁵.

AI systems learn from data, and data reflects the world it came from, including its gaps and prejudices. A 2018 study found that two widely used face datasets were 79.6% and 86.2% made up of lighter-skinned faces². A model trained on data like that has seen far fewer examples of everyone else.

The US standards body NIST groups AI bias into three types. Systemic bias comes from institutions and society, such as historical discrimination in who got loans or jobs. Statistical and computational bias comes from data that does not represent the population. Human bias comes from how people build, read and act on the system's outputs. NIST stresses that any of these can occur without anyone intending to discriminate¹.

The best-known evidence came from the 2018 Gender Shades study by Joy Buolamwini and Timnit Gebru. Testing three commercial gender classification systems, they found error rates of up to 34.7% for darker-skinned women, against a maximum of 0.8% for lighter-skinned men².

Bias also appears in generative AI. NIST's 2024 profile for generative AI notes that when asked for images of CEOs, doctors, lawyers and judges, current text-to-image models under-represent women and racial minorities³.

Bias is not the same as error. All models make mistakes; bias means the mistakes fall unevenly on particular groups.

Example

In 2020 the Court of Appeal ruled that South Wales Police had not done enough to meet its public sector equality duty, because it had never checked whether its automated facial recognition software had an unacceptable bias on grounds of race or sex. The manufacturer would not share the details needed to test it⁴.

Why it matters

Biased systems can deny people jobs, loans or fair treatment by police, often without them knowing an algorithm was involved. Because AI works at scale, one biased model can affect millions of decisions.

Questions people ask

What causes algorithmic bias?

NIST names three sources: systemic bias from society and institutions, statistical bias from unrepresentative data, and human bias in how systems are built and used. None of them requires anyone to intend harm¹.

Can algorithmic bias be fixed?

It can be reduced, not simply removed. Common steps include testing accuracy separately for different groups, improving training data and keeping a human in the loop. NIST notes that a system with balanced predictions can still be unfair in other ways⁶.

Is algorithmic bias illegal in the UK?

There is no specific law on it, but existing law applies. The Equality Act 2010 bans discrimination, and public bodies have an equality duty. The Court of Appeal used that duty in 2020 to rule against South Wales Police's use of facial recognition⁴.

Related terms

Sources

Back to the glossary