From startups to Big Tech, everyone loves to tout their strategy for leveraging machine learning (ML). These companies promise ML is our 21st century savior; it will not only liberate us from the tedious drudgery of administrative tasks but will arm us with near-perfect predictions.
But the technology has a dark side, one that few companies adequately acknowledge and even fewer are equipped to handle. Underneath MLs sexy facade is essentially a self-reinforcing statistical algorithm.[1] The system does not bother with causation, but rather finds predictive correlations within data sets it’s given.[2] It then makes judgements based on its programming and subsequently its own, unchecked learning.[3] And therein lies the problem with ML: it uses biased data to make biased judgements and then further encodes that bias into its opaque system.[4]
Nowhere is this danger illustrated more perfectly than in Amazon’s quest to streamline its hiring efficiency.
The Importance of Machine Learning At Amazon
In three years, Amazon’s headcount has more than tripled to 575,000+ employees.[5] Napkin math suggests this translates into Amazon engaging with 1600+ applicants every day.[6] The company needs an efficient process to minimize throughput time per application and optimize recruiter takt time to meet internal hiring demands. Hiring process improvement could result in millions of dollars in cost savings, while ensuring the company hires the talent needed for continued domination.
In 2014, Amazon put together a team of engineers charged with using machine learning to improve their hiring process.[7] A system was built that could analyze applications and assign each applicant a 1-5 star rating, much like that on Amazon.com.[8] Executives reportedly believed this project to be the “holy grail of hiring” and would alleviate a need for human involvement.[9]
Amazon was not alone in these aspirations or efforts. 55% of U.S. human resources managers said ML “would be a regular part of their work within the next five years.”[10] Notable companies like Goldman Sachs and Hilton are building their own ML-powered hiring systems.[11] While Amazon’s efforts are not unique, its status as one of the world’s most valuable companies often leads others to look to Amazon as an example of best-in-class. This influential position makes their problems with ML all the more troublesome.
Trouble At Amazon
In October 2018, news broke that Amazon’s hiring algorithm was sexist: it preferred male candidates to women by an order of magnitude.[12] This result underscores the dark side of ML:
Biased Data
Machine learning is dependent on the data its fed and centuries of sexism, racism, homophobia and more have corrupted every data set.[13] Moreover, the communities that often suffer from bias are typically communities that are underrepresented in data sets, leading to a disproportionate level of influence.[14] This is particularly troublesome for an application like Amazon’s where an unchecked machine further harms marginalized populations because those populations have already been harmed.
Correlation Not Causation
ML focuses on finding correlations, not causations.[15] This means that ML is creating heuristics, the bedrock of stereotyping, to allegedly predict the future.[16] For example, Amazon correlated women with lower performance from its own biased data.[17] It then was able to identify applicants as women from linguistic patterns and rate them lower.[18] This exemplifies how ML’s correlation-centric approach makes it particularly capable of reinforcing bias, an effect that is only worsened by MLs unchecked feedback loop.[19] As former Google AI and search head John Giannandrea puts it, “When these types of human mistakes become baked-in parts of an AI…we’re not creating artificial intelligence: we’re just obfuscating our own flawed observations inside of a black box.”[20] This black box feature is worrisome given few know that are being judged by ML and even fewer have the technical expertise to challenge suspected bias.[21] Complicating this are existing laws that allow criminal prosecution of those who test hiring websites algorithms for discrimination. [22]
Amazon’s Solution
After four years of work, the Amazon repositioned its ML team to focus on diversity while using a “much-watered down version” of the recruiting engine to work through rote administrative tasks. [23]
But these actions are far from sufficient. The company must build intersectional teams that include activists and psychology experts with deep experience with bias. Furthermore, Amazon should build a publicly available database that codifies all potential biases across datasets and develop open source ML algorithms that can help correct for this. Perhaps most importantly, Amazon should be vigorously supporting social justice work to help stop the problem at its source, and improve the world in the process.
But will these measures be enough? Is it okay that ML only relies on correlation, not causation? Is ML actually a useful predictor or is it just a powerful stereotyper? As we transition to a ML-first world, these are questions that demand our rigorous attention, debate, and ultimately, action.
(800 Words)
[1] Alpaydin, E., 2014. Introduction to machine learning, Cambridge, MA: The MIT Press.
[2] Ibid., Accessed November 2018.
[3] Ibid., Accessed November 2018.
[4] Vanian, J., 2018. Unmasking A.I.’s Bias Problem. Fortune, accessed November 2018.
[5] Anon, 2018. Amazon.com Announces Fourth Quarter Sales up 38% to $60.5 Billion. Amazon.com, Inc. – IR., accessed November 2018.
[6] Assuming 140K jobs added each year. 350 working days each year. 4 applicants needed for every hire.
[7] Dastin, J., Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, accessed November 2018.
[8] Ibid., Accessed November 2018.
[9] Ibid., Accessed November 2018.
[10] Dastin, J., Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, accessed November 2018.
[11] Ibid., Accessed November 2018.
[12] Dastin, J., Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, accessed November 2018.
[13] Waxman, A. (2018) ‘AI can help banks make better decisions, but it doesn’t remove bias’, American Banker, 183(108), p. 1, accessed November 2018.
[14] “AI: The Issue of Bias.” Managing Intellectual Property, 17 Sept. 2018, pp. N.PAG-N.PAG. Business Source Complete, EBSCOhost, accessed November 2018.
[15] “AI: The Issue of Bias.” Managing Intellectual Property, 17 Sept. 2018, pp. N.PAG-N.PAG. Business Source Complete, EBSCOhost, accessed November 2018.
[16] Pearl, J., 1985. Heuristics: intelligent search strategies for computer problem solving, Amsterdam, NL: Addison-Wesley.
[17] Dastin, J., Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, accessed November 2018.
[18] Ibid., Accessed November 2018.
[19] “AI: The Issue of Bias.” Managing Intellectual Property, 17 Sept. 2018, pp. N.PAG-N.PAG. Business Source Complete, EBSCOhost, accessed November 2018.
[20] Dastin, J., Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, accessed November 2018.
[21] Knight, Will. “The Dark Secret at the Heart of AI.” MIT Technology Review, 11 Apr. 2017.
[22] Bhandari, E. & Goodman, R., 2017. ACLU Challenges Computer Crimes Law That is Thwarting Research on Discrimination Online. American Civil Liberties Union, accessed November 2018.
[23] Dastin, J., Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, accessed November 2018.
Image Source: www.arstechnica.com