Algorithmic bias

Joseph Reagle

Our question(s)

  • Are algorithms biased?
  • Why and how?
  • What to do?

Racial bias in ads

Latanya Sweeney

“Latanya Sweeney, Arrested? (1) Enter name and state (2) Access full background. Checks instantly. www.instantcheckmate.com”

  • Google searches implying arrest
    • black-sounding names: 60%
    • white-sounding names: 48%
    • “black-sounding names” 25% (60-48/48) more likely (Solon 2013)

Response

  • Google: “AdWords does not conduct any racial profiling. We … will not allow ads that advocate against an organisation, person or group of people. It is up to individual advertisers to decide which keywords they want to choose to trigger their ads.”
  • Instant Checkmate: “[we] would like to state unequivocally that it has never engaged in racial profiling in Google AdWords. We have absolutely no technology in place to even connect a name with a race and have never made any attempt to do so.”

… people are …

black people are… white people are…

(A click — vs. link — version of the Bush GoogleBomb)

Joy Buolamwini on bias

Why? Sill true??

Can you find a “live” version of algorithmic bias now?

Bias in images

Shirley cards (1940s)

Lenna JPEG (1973)

Three “Shirleys” (1995)

Google Pixel 6 “Real Tone”

Bias in AI

Diffusion bias

See if you can discover any biases.

Tay Tweets racism

ChatGPT wouldn’t write….

  • a poem about Trump (but will for Biden)
  • a story about Trump beating Biden (but will for Clinton beating Trump)
  • about COVID vaccine having harmful side effects
  • a critique of drag-queen story hour (Hochman 2023)
  • Joe & Isabel: current state of algorithms?

Gemini won’t generate white people

(Crimmins 2024)

Weapons of Math Destruction

[W]idespread [M]ysterious [D]estructive

What is an algorithm?

An algorithm often requires a model.

What is a “model”?

nothing more than an abstract representation of some process, be it a baseball game, an oil company’s supply chain, a foreign government’s actions, or a movie theater’s attendance (O’Neil 2016, p. 18)

How do you create a model?

we make choices about what’s important enough to include, simplifying the world into a toy version that can be easily understood and from which we can infer important facts and actions (O’Neil 2016, p. 20)

Healthy vs toxic model of WMD??

What’s the difference between baseball and law enforcement?

  • baseball
    • transparent
    • rigorous
    • relevant
    • continually updated
  • law enforcement
    • opaque (e.g., LSI-R)
    • poisonous assumptions
    • pernicious feedback loop
    • harmful because of (a) scale and (b) effect

Arms race: Going to college

Why rankings are a racket

“How to Game the College Rankings”

Northeastern University executed one of the most dramatic turnarounds in higher education. Its recipe for success? A single-minded focus on just one list. (Kutner 2014)

What is a successful college?

[Scores] couldn’t measure learning, happiness, confidence, friendships, or other aspects of a student’s four year experience.

President Lyndon Johnson’s ideal for higher education “a way to deeper personal fulfillment, greater personal productivity and increased personal reward” didn’t fit into their model. (O’Neil 2016, p. 52)

What is a proxy for success?

Instead they picked proxies that seemed to correlate with success. They looked at SAT scores, student-teacher ratios, and acceptance rates… They calculated the percentage of living alumni who contributed money to their alma mater, surmising that if they gave a college money there is a good chance they appreciated the education there. (O’Neil 2016, p. 52)

Consequences??

  • Monomania
  • Gaming/cheating
    • student admission scandals
    • faux faculty
  • Tuition increases

Monomania

Colleges were like different types of music, or different diets. There was room for varying opinions, with good arguments on both sides. Now the vast reputational ecosystem of colleges and universities was overshadowed by a single column of numbers (O’Neil 2016, p. 53)

Cheating/gaming

  • Baylor paid admitted students to retake SAT
  • Bucknell, McKenna, and Iona sent false data
  • Northeastern pushed lower SATs and bigger classes to spring semester (O’Neil 2016, p. 58)

Faux faculty

the Saudi university had contacted a host of mathematicians whose work was highly cited and had offered them $72,000 to serve as adjunct faculty (O’Neil 2016, p. 62)

Admissions consultancies

A four-day “application boot camp,”… costs $16,000 (plus room and board)…. the high school juniors develop their essays, learn how to “ace” their interviews, and create an “activity sheet” the sum up all the awards, sports, club activities, and community work that admission officers are eager to see.

the college admissions game, while lucrative for some, has virtually no educational value (O’Neil 2016, p. 64-65)

Brittle about students

  • don’t risk taking an at-risk student (they could hurt your retention)
  • don’t risk taking an excellent student (they could hurt your yield)

Cost wasn’t in the model

[Colleges] had a commandment to maximize performance in fifteen areas, and keeping costs low wasn’t one of them…

Higher Ed ↑500% (4x inflation) from 1985–2013

to attract top students, colleges … have gone on building booms, featuring glass walled student centers, luxury dorms, and gyms with climbing walls and whirlpool baths (O’Neil 2016, p. 60)

Obama’s fix

Focus on cost and employment, which led to:

  • cuts to humanities and arts
  • counting those who don’t respond to a survey as employed
  • counting baristas with $150,000 in student loans as gainfully employed
  • hiring grads as temps during employment reporting period

Result

the government capitulated. And the result might be better. Instead of a ranking, the Education Department released loads of data on a website.

Think of it: transparent, controlled by the user, and personal. You might call it the opposite of a WMD. (O’Neil 2016, p. 67)

How has this affected Northeastern?

  • Puts pressure on students to graduate in 4 years, or to take 4+1 degree.
  • Increasing class sizes because USN drop <=19 class size.

NYT’s “Build your own”

Colleges abandoning U.S. News Rankings

Conclusion

Wrap up

With peers, write down three techniques for curbing “weapons of math destruction.”

Review

What were your techniques for curbing “weapons of math destruction”?