• Joy Buolamwini is an MIT researcher working to compel organizations to make facial recognition software more ethical and inclusive.
  • The AI that runs facial recognition software learns from data during its development, and its accuracy in assessing faces reflects that data set.
  • After Buolamwini shared research with IBM about its software’s poor scores in assessing darker male and female faces, IBM developers were able to drastically improve accuracy scores through adjustments.
  • This article is part of Business Insider’s ongoing series on Better Capitalism.

When we talk about algorithms and automation, we can’t assume that handing responsibilities over to a machine will eliminate human biases. Artificial intelligence, after all, is constructed and taught by humans.

MIT Media Lab researcher and Algorithmic Justice League founder Joy Buolamwini has made it her mission not only to raise awareness of bias in facial recognition software, but also to compel companies around the world to make their software more accurate and to use its capabilities ethically.

Buolamwini coined the term “the coded gaze.” She says the coded gaze is a “reflection of the priorities, the preferences, and also sometimes the prejudices of those who have the power to shape technology,” she told an audience at the World Economic Forum’s 2019 annual meeting in Davos, Switzerland.

There are real stakes here. As she noted in a viral TED Talk and a New York Times editorial, it’s one thing to have Facebook confuse people when analyzing a photo, but another when law enforcement or a potential employer is utilizing such software.

Buolamwini has Ghanian heritage, and as a graduate student, she found that facial recognition software she was researching could detect the faces of light-skinned classmates but not her own, darker skin. She showed the Davos crowd a shot from her TED talk, in which she demonstrated the way that same software recognized her face when she wore a plain white mask over it.

Different companies use different software for facial recognition, but the AI involved learns through images fed to it during its development. If it's shown primarily white, male faces, it will become an expert at identifying white, male faces and can pick up subtle details; meanwhile, it will struggle to identify or differentiate faces with different skin hues, and will have a more difficult time assessing female features.

Buolamwini and MIT colleagues embarked on a thorough assessment of leading facial recognition programs - those from Microsoft, Face++, and IBM - to see how they fared. There were differences in exactly how accurate each program was, but all three generally had the same order of best to worst accuracy: lighter male, lighter female, darker male, darker female (Face++ had a .1% increase in accuracy for darker male than lighter male).

Before making her research public, Buolamwini sent the results to each company. IBM responded the same day, she said, and said developers would address discrepancies. She assessed IBM's updated software last year and found a notable improvement. The accuracy for correctly assessing darker males went from 88.0% to 99.4%, for darker females from 65.3% to 83.5%, and for lighter females from 92.9% to 97.6%, and lighter males stayed the same at 97.0%.

"So for everybody who watched my TED Talk and said, 'Isn't the reason you weren't detected because of, you know, physics? Your skin reflectance, contrast, et cetera,' - the laws of physics did not change between Dec. 2017, when I did the study, and 2018, when they launched the new results," Buolamwini said. "What did change is they made it a priority."

Buolamwini then recognized that while software primed for lighter males presented its own problems, even perfect software was at the whim of its handlers.

She noted an Intercept investigation published last summer that found IBM had worked with the New York City Police Department for years on a surveillance project utilizing facial recognition, which was never made public while in use. (The NYPD told the Intercept it never used the software's ability to distinguish skin color, IBM declined to comment on the project, and the NYPD ended its partnership with IBM in 2016.)

She then pointed out how hiring software like HireVue, a company that makes a program that allows large employers like Unilever and Hilton Worldwide to rapidly assess many job applicants' video interviews, also utilize facial recognition software. Boulamwini argued that if used without sufficient care, AI hiring programs could reinforce existing traits in a role rather than correct for them. (HireVue told Business Insider that testing for bias with the intention of eliminating it is an integral aspect of the way its software is used.)

"So it's not just the question of having accurate systems," she said. "How these systems are used is also important." It's why she launched the Safe Face Pledge, with four commitments: show value for human life, dignity, and rights; address harmful bias; facilitate transparency; and embed commitments into business practices. It launched with three signatories, but has yet to land a major corporation.

Facial recognition software is here to stay, and has potential for useful applications like protecting users from identity theft. As Buolamwini's research suggests, it poses less of a threat to citizens' rights the more accurate it is, and as its use becomes more widespread, companies must take the ethics around its use very seriously.

You can see Buolamwini's full presentation at WEF's website »