On May 7, the Digital Data Design Institute at Harvard hosted Leading with AI: Exploring Business and Technology Frontiers. The conference featured breakout sessions where HBS staff and industry experts discussed specific and specialized topics. The breakout session on The A New Paradigm for Skill Development: A Large-Scale BCG Experiment featured speakers from he Boston Consulting Group (BCG): François Candelon, Global Director of the BCG Henderson Institute; Lisa Krayer, Principal; and Daniel Sack, Managing Director and Partner of BCG X.
Key Insight: Beyond the Jagged Frontier
“[We wanted] to see whether our consultants [with] no or limited data science capabilities…could actually solve data science issues [using ChatGPT].“
François Candelon
In 2023, Boston Consulting Group (BCG) and the HBS Digital Data Design Institute conducted an experiment on the use of AI in skill development (see the working paper, Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality). Briefly, this was a scientific study with real employees that looked at how AI can automate some tasks to improve performance. One result was a new term, the “jagged technological frontier,” where some tasks are easily done by AI, while it cannot do others that are similar in difficulty level.
In a more recent study, BCG wanted to look beyond these results to see if AI could help them up-skill volunteer consultants without backgrounds or experience related to the study tasks. François Candelon, Dan Sack, and Lisa Krayer presented 10-day-old preliminary results of this study.
Key Insight: Experimental Structure and Early Results
“[C]an we give people tooling that helps them get better or more efficient with the tasks they know how to do? …[C]an we understand what they can do with something they have no idea what they’re doing?”
Dan Sack
For this study, 525 consultants and 40 data scientists in different age groups from around the world worked on three tasks:
- Predictive analytics (using historical data to predict games in the future)
- Data cleaning (non-programmers assessing the accuracy of data)
- Statistical analysis (identifying when ChatGPT analysis was incorrect)
The data scientists were the baseline, one group of consultants used ChatGPT to work on the tasks, and the other group of consultants did not use ChatGPT to work on the tasks. Both consultant groups received some training on how to use tools available to them:
- Chat GPT group: Prompting best practices, few-shot prompting, chain-of-thought prompting, and Code Interpreter tool
- No ChatGPT group: Stack Overflow, Khan Academy, Python documentation, and search strategies
The study team segmented results based on consultants’ previous experience with analytics, data analysis, and learning models to avoid skewing the results. But some early results showed that consultants with no programming experience achieved 17% of what a data scientist could accomplish within the given 90-minute time frame on a data cleaning task. And the team found that with statistical analysis, everyone got a “big boost,” even those without experience. The predictive analytics task was more difficult to measure because there was no “right answer.” But they found that consultants with the least experience had better outcomes on this task, and want to find out why.
Key Insight: Learning and Development
“[W]e want to understand how this works for learning and development as well, not just can people do this, but do they learn something while they’re doing it?”
Dan Sack
After the study, the team tested participants’ retention and found similar levels across the segments. They want to understand the learning curve over time, beyond the 90-minute test period. Dan stressed that the study was not designed to teach participants something beyond the one-time tasks. The multistep, iterative nature of the tasks and ChatGPT suggest the need to design longer-term tasks to measure learning and retention. And future tests also must design interactions between teams and systems because AI often does not work “right out of the box in one shot” and must iterate to find answers.
Key Insight: Confidence, Overreliance, and Adoption
François Candelon
“[B]y pushing the adoption, basically you are igniting a positive, virtuous cycle…the more you use, the more you trust, and the more you trust…the more you use…the worst day of AI is day one, so it’s really important to do that and I strongly believe that AI transformations are of a different nature than the others because it has a massive impact on your professional identity, competency, sense of autonomy, sense of belonging, and/or relatedness.”
The team discussed the risk of overconfidence leading to overreliance on AI, and skepticism that makes people and companies avoid it. Companies must understand the jagged frontier and when AI is helpful and when it is not. Increasing engagement with AI can help drive adoption by building confidence in themselves and the tool. Companies should be intentional when rolling out AI tools and clarify why it is valuable. They must consider redesigning their processes and ensure proper supervision to make sure the quality of the output is high.
Frameworks
Change Management
“What was interesting in our previous experiment is that when we are asking the consultants whether they were feeling that they were afraid, because maybe strategy or problem-solving will be done, they were saying, we’re not that afraid because we believe there are some other things to do where we are still, for instance, change management becomes very important.”
François Candelon
Adoption of AI involves important work on change management and company culture. For BCG and many companies, AI can be a competitive advantage, but it also involves looking carefully at workforce planning and talent strategies—recruiting, interviewing, learning and development, and promotions. Companies must realistically assess these systems’ capabilities. Part of the change management process is educating employees on how to work effectively with AI and delegating work to AI for first-draft and peer-editing work.
In François’ view, “AI can help culture…we believe that we need to change culture to adapt to AI, but really I would say it’s almost the other way around. AI can help change culture in a positive way” through improved employee professional identity (competence, autonomy, and connection) and morale. As François noted, “[it’s] not about optimizing the technology, it’s about optimizing the way humans use this technology.”
Implementation Questions
“It’s incredibly important to listen to people and not just make decisions. And talk to people about how, because we don’t know how this is going to impact people’s lives and we won’t know until we talk to them and truly take that initiative to listen.”
Lisa Krayer
The session participants and the team surfaced many questions around implementation of AI, including:
- How do you teach discernment around AI results?
- When should you be confident (or not confident) in results?
- How do you supervise AI work when you’re not trained in AI?
- How do you monitor negative AI information and experiences?
- How should employees be hired or promoted when their work is or is not augmented by AI?
- How will data scientists’ jobs change? How will employees’ jobs change if AI shifts capabilities and responsibilities?
- When you implement AI, do you need to change workflows and breaks to ensure employees still interact in person? How can employees share in the benefits of productivity gains?
Meet the Speakers
François Candelon is currently a Partner at Seven2, a French private equity firm. Previously, he was a Managing Director and Senior Partner of Boston Consulting Group and Global Director of the BCG Henderson Institute, BCG’s strategy think tank. François earned his MSc, Economy from École Polytechnique, a degree from Mines Paris – PSL, and a DEA from Université Paris Dauphine – PSL.
Lisa Krayer is a Principal at Boston Consulting Group, where she has conducted research into the impact of generative AI on individuals and businesses for the BCG Henderson Institute. Lisa obtained her BS in Physics from UC San Diego, and her MSc in Electrical and Electronics Engineering and PhD in Electrical and Electronics Engineering from the University of Maryland.
Daniel Sack is a Managing Director and Partner of BCG X, BCG’s tech build and design division, building digital products based on machine learning and AI. Dan is BCG’s Global Data Science Chapter lead, and he leads BCG X in the Nordics. Dan earned his MBA from Stanford University Graduate School of Business and his BE in Mechanical Engineering from Dartmouth College.
Additional Resources
François Candelon:
- Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (research paper)
- The Rise of AI-Powered Companies (Inspiring the Next Game) (book)
Lisa Krayer:
- How People Can Create—and Destroy—Value with Generative AI (article)
- The CEO’s Guide to the Generative AI Revolution (article)
Daniel Sack: