Earlier this week we hosted our most recent Mayfield CXO Insight Call: “Rationalizing the AI Agenda,” where Erik Brynjolfsson, the Jerry Yang and Akiko Yamazaki Professor and Senior Fellow at Stanford University and Andrew McAfee, Principal Research Scientist at MIT, shared their insights on how both business and technology leaders are rationalizing their AI agendas.
Here were some of the key learnings:
Over the last couple of years, consumers and technology leaders alike have been captivated as milestone after milestone have been shattered by AI, ranging from winning math olympiads to achieving huge advancements in image recognition. Unfortunately, the flipside of that has been continued disappointment surrounding the translation of these amazing accomplishments into real business value.
By now, most companies have moved past the exploratory phase of “Should we adopt AI?” and even beyond the initial pilot stages. Despite this progress, several key questions still linger in the minds of many:
A recent HAI report came out of Stanford, the 2024 AI Index Report, which tries to answer some of these questions and act as a centralized repository for ongoing progress and benchmarks in AI. It started off relatively thin and now it’s over 500 pages long. Another good report on the topic is Artificial Intelligence and the Future of Work, which came out of the National Academies. In summary: everyone has been blown away by AI’s technological progress, but the changes in productivity, the economy, and the business value have all been slower.
There are reasons to expect that this outcome is somewhat normal. This actually happened with earlier technologies like electricity, or the steam engine. And one thing to bear in mind is that companies which were on top at the start of these big transitions usually didn’t stay that way. Significant turbulence is part and parcel with the adoption of new technologies, and a lot of creative destruction tends to occur naturally as a part of this process. The question that’s top of mind for many right now is whether or not we’re going to see that movie repeat itself today, or if the successful incumbents of the 20th century will maintain their advantage. Erik and Andy have even started a company, Workhelix (which just raised its Series A), in order to try and fill this need: giving companies the tools necessary to understand whether or not their AI is effectively implemented.
So what concrete things can organizations do in order to accelerate the time to value of AI? One idea to consider is implementing a task-based approach. It’s painful to watch companies spend millions or even hundreds of millions on projects that are intellectually interesting, but which don’t accomplish much for the business. That’s where having a task-based approach comes in. The goal should be to identify the specific tasks that create value instead of looking at the entire organization or across employee roles. Much of the approach to AI today has been: At which job will AI be the most effective? That may not be the right idea. Tasks are the atomic unit of work, and every occupation is merely a bundle of these tasks which can be divided in different ways.
Today, the Bureau of Labor Statistics has a taxonomy of around 18,000 tasks. A radiologist, for example, does 27 tasks, a call center worker, 23. You can break down what everyone does as a matter of course within their various roles and responsibilities. The key is that for each of these tasks, teams can evaluate whether or not there’s a tool out there or a technique that can help. We all know, for example, that if you need to write a memo, a large language model can help. And while that same LLM will not help you lift a box or drive a taxi, there are other technologies which could be considered. Erik and Andy’s new company, Workhelix, has scored over 250,000 individual tasks and rolled them up for the enterprise after multiplying them by a value indicating their importance. If companies take the ability of technology to help with any given task and multiply it by the value created, they can start to develop a roadmap that is really business oriented.
There are easy wins today that almost every organization can benefit from. Coding is a huge win and call centers are another area where there’s been a lot of progress. But every company, across every industry, will have a unique footprint in terms of where they can derive the most value from technology. It’s crucial to always use real value being created as the north star, rather than simply playing around with new tools that have great benchmarks.
When taking such an approach, however, it’s always important for leaders to bear in mind that you don’t just want to optimize for individual tasks, which could perhaps create incremental improvement without consideration of the entire outcome. Processes are ultimately bundles of tasks – so instead of rolling up to the job level, you can roll up to the process level and consider how the most important processes can be impacted by the AI toolkit. You almost never see a technology capable of doing all of the tasks in a particular job. Technology tends to be capable of doing some parts of the job, which is where managers need to come in and think. How can the company reconfigure the way that a process or an occupation is done in order to take advantage of the strengths of new technology? It’s not simply unplugging someone and replacing them with technology – which is too often the mindset that executives try to get away with.
Further, even with successful implementations of new technologies, teams will always need to keep an eye out for process bottlenecks. You may not care too much how fast a microprocessor is on your laptop if you’re unable to type or write all that much faster as a result. These bottlenecks only become visible when new technology speeds up some of the components in a process, which creates the opportunity to model how all these given tasks relate to one another. Understanding these dependencies is the first step to understanding where the bottlenecks are. It almost goes without saying that you need to focus on those bottlenecks if you want to speed things up, not speed up the parts that are already unconstrained.
Of course, all of this rationalization does require a little bit of top-down thinking. Ideally, there’s a CIO or a CTO who can take ownership of such a process, but ultimately buy-in will need to happen up and down the organization, just like with any other major technology initiative. One way to help with this is emphasizing that new technology is arriving in order to help and support workers, not replace them. There’s been a lot of fear drummed up around AI, and employees may believe these tools are so powerful that they’re able to wholesale replace thousands of their peers. And while in a few cases you can, most of the time AI is merely a tool for augmentation.
Furthermore, one trap executives tend to fall into while communicating, is emphasizing the cost savings around AI. While there are definitely many opportunities for cost savings, this limits creative thinking around some of the other benefits. It’s important for companies to think about the full suite of KPIs – How can you improve customer satisfaction or customer sentiment? How can you reduce employee turnover? How can you improve the quality of code (not just the output)? Costs are one of the easiest things to measure, so that’s often where the focus tends to land. There’s nothing wrong with saving costs, but focusing on that to the detriment of everything else causes leaders to look over some of the bigger wins. Obsessing about costs can create a zero-sum mentality, which makes adoption harder. Ultimately, the biggest gains in both society as well as large organizations tend to come from doing new things, and providing new goods and services or new kinds of value, not simply doing the same things at a lower cost.
As an example: look at generative AI in materials science. When the technology is combined with the expertise of researchers, it’s able to surface more candidate molecules and materials for the scientists to investigate, and a higher percentage of those materials turn out to be commercially viable. As a result, there’s measurable downstream product innovation coming out of the implementation of AI technology.
Eric and his colleagues published “Generative AI at Work” in the Quarterly Journal of Economics which focused on the impact that generative AI has had on companies through the lens of customer service. The findings:
These kinds of deep reports are very multidimensional, and it’s important for companies to bear in mind that this is how they should be approaching the analysis of their own AI implementations. You have to look at many different dimensions of interest and figure out whether or not a new technology is helping move the needle in a significant way. Typically, the best option isn’t just turning over the keys to an agent and having it answer all the questions. Those questions may be relatively complicated, and we humans, for now, are still better at dealing with those. So having a partnership where humans and machines work together is just dramatically more successful than attempts to have it be entirely automated. More often than not, humans + machines outperforms machines alone.
It’s crucial that the metrics of productivity generated by AI are legible for other executives and the board to absorb – so don’t be shy about including a wide range of data. In the example above, it’s not just cost, it’s productivity, it’s fluency, it’s employee retention, and it’s customer satisfaction. It’s tacit knowledge. Many companies are not thinking creatively or using their data effectively. The CFO was a very powerful position in organizations throughout the 20th century, so companies have overindexed on quantitative metrics in general. LLMs, which have the word language baked into them from the start, often lead to more ambiguous ways of evaluating data. This qualitative data can nonetheless be very impactful.
There are some cautionary tales here as well. Super smart people looked at how to implement machine learning in social media and discovered that it could maximize engagement, clicks, views, etc. Companies did a great job at maximizing those metrics, but in some cases, this was at the expense of long-term satisfaction and value, which ultimately turned off a lot of people. So bear in mind that it’s not always about overemphasizing the short-term, easy-to-measure metrics, you want a broader collection of KPIs, including some which are measured over the medium to long term, and some which maybe aren’t as easy to quantify.
Bear in mind that whether or not the full value of AI can be exploited today, it’s very important to get started on a roadmap. The gap between the leaders and laggards in the space is significant and continuing to grow. Companies like Walmart are already emerging as premier AgTech (agentic) leaders and the downstream impacts could be enormous.
GE was delisted from the Dow Jones average, after being a part of the original group of companies that comprised it. Boeing is in trouble. Netflix has a value of close to $500B, and Warner Brothers is only worth $20-25. It’s not just about AI – it’s about technology as a concept. Every company in business today must be a technology company in order to thrive. As AgTech and other emerging areas start to show early results, if companies don’t experiment and figure out where their ROI is, they will land on the laggard side of the equation. This gap isn’t growing because people don’t have access to these new technologies, it’s growing because they can’t figure out how to use them effectively. It’s crucial to stay data-driven about figuring out where the value is, and translating that value into benefits.
Traditional organizations tend to spend a huge amount of time planning things out, ensuring that they anticipate all the contingencies, and trying to prevent things from going wrong. As a result, they are very slow, and then things still go wrong. By comparison, many of the more modern, “geekier,” companies have nailed down a more iterative approach of doing things (even things which might be wrong), gathering feedback, and trying something new – on the fastest possible cycle. There are exceptions to this, of course, you don’t want to blow up a rocket with someone standing on it. But in general, this “build, test, learn, and gather feedback” approach has been widely successful across nearly every industry. If older companies stay in the planning-heavy world, and the geeks arrive at their doorstep, they’re going to find themselves in trouble.
That being said, as an IT leader, if you’re selling a project to a business executive, bear in mind that they may not be all too familiar with these new technologies. Why would the CMO of your organization want to sign up for a huge boost in metrics (that they’re responsible for) if they don’t have confidence in, or familiarity with, these new technologies? Instead of trying to force over commitment on these executives – maybe start small, with pilot projects. Can you set up a study and watch what happens? You want fast feedback, the same as any other historic tech innovations. If the executive is happy with the results, there’s now a basis upon which to scale things up.
Erik Brynjolfsson is the Jerry Yang and Akiko Yamazaki Professor and Senior Fellow at the Stanford Institute for Human-Centered AI (HAI), and Director of the Stanford Digital Economy Lab. He also is the Ralph Landau Senior Fellow at the Stanford Institute for Economic Policy Research (SIEPR), Professor by Courtesy at the Stanford Graduate School of Business and Stanford Department of Economics, and a Research Associate at the National Bureau of Economic Research (NBER).
One of the most-cited authors on the economics of information, Brynjolfsson was among the first researchers to measure productivity contributions of IT and the complementary role of organizational capital and other intangibles. He has done pioneering research on digital commerce, the Long Tail, bundling and pricing models, intangible assets and the effects of IT on business strategy, productivity and performance.
Brynjolfsson speaks globally and is the author of nine books including, with co-author Andrew McAfee, best-seller The Second Machine Age: Work, Progress and Prosperity in a Time of Brilliant Technologies, and Machine, Platform, Crowd: Harnessing Our Digital Future as well as over 100 academic articles and five patents. He holds Bachelors and Masters degrees from Harvard University in applied mathematics and decision sciences and a PhD from MIT in managerial economics. His papers can be found at https://www.brynjolfsson.com/research
Andrew McAfee (@amcafee) is a Principal Research Scientist at the MIT Sloan School of Management, co-founder and co-director of MIT’s Initiative on the Digital Economy, and the inaugural Visiting Fellow at the Technology and Society organization at Google. He studies how technological progress changes the world. His next book, The Geek Way, will be published by Little, Brown in 2023. His previous books include More from Less and, with Erik Brynjolfsson, The Second Machine Age.
McAfee has written for publications including Foreign Affairs, Harvard Business Review, The Economist, The Wall Street Journal, and The New York Times. He’s talked about his work on CNN and 60 Minutes, at the World Economic Forum, TED, and the Aspen Ideas Festival, with Tom Friedman and Fareed Zakaria, and in front of many international and domestic audiences. He’s also advised many of the world’s largest corporations and organizations ranging from the IMF to the Boston Red Sox to the US Intelligence Community.
McAfee and his frequent coauthor Erik Brynjolfsson are only people named to both the Thinkers50 list of the world’s top management thinkers and the Politico 50 group of people transforming American politics.
Workhelix brings together cutting-edge research and comprehensive data to assess companies’ Generative AI opportunities. Its software creates an interactive, comprehensive roadmap tailored to your organization.
