Author: Shelby Golan

Today we welcome Yasir Anwar, former CTO and Chief Data Officer for Williams Sonoma, to The CXO Journey to the AI Future podcast.

Yasir Anwar is an innovation enthusiast who is an Entrepreneur at heart and a Technologist at execution. He considers himself on a D.I.E.T (Disruption; Innovation; Execution; Technology Excellence). To keep the balance between solving technical issues and shaping an overarching strategy, he strives to switch roles between being an engineer, architect, and C-Level Executive as needed. He believes this keeps himself humble, realistic and truly connected with his teams.

Most recently he was the Chief Digital and Technology Officer for Williams Sonoma Inc. leading all digital, technology, and IT functions. Before this, he was the first CTO for Macy’s, where he drove the growth of e-commerce channels, and omni-channel capabilities and led the merger of the previous digital and CIO organizations under one mission and culture.

Yasir is also a Lean practitioner at scale and established the LeanLabs culture at Macy’s which grew from one LeanLabs team to 40+ teams contributing significantly to an increase in revenue, cultural agility, and innovation.

Question 1: Current Job: Could you tell us a little bit about your background and how you ended up where you are today?

This career is what I’ve always wanted. I’ve tried to work towards being both an entrepreneur and an execution person, and it has taken a lot of work to get to where I am today.

I started as a core engineer in C++, Java,…all those languages and databases. And I’ve always believed that everything I’m doing must make a business impact (whether it’s internal or customer-facing revenue development). It can’t just be a shiny object or an academic exercise.

So, I’ve tried to make an impact via tech innovation, and more importantly, always measure it. I’ve always focused on how to better devise mechanisms, culture, and frameworks in order to facilitate continuous measurement.

That has been the core thread of the career I’ve developed. In all the leadership roles I’ve had, whether at Walmart, Macy’s, or William Sonoma, making an impact was ultimately accomplished by building great teams and a great culture, while still being capable of partnering with the broader ecosystem.

Question 2: Generative AI: Of course, here in the valley AI is all the rage. San Francisco has several new startups that are AI-focused. But you’ve been through quite a few technology journeys of your own. How much priority do you think we should be putting on this?

I think generative AI is critical to the growth, evolution, and innovation of any business. If you look at it from a time series function, last year was a year where companies and businesses should have invested in exploring and understanding the art of the possible in both the near-term and the long-term.

2023 was a year of exploration for most of the people who are not producing core AI solutions. I envision that over the long-term, generative AI will drive a paradigm shift of resources, mind share, and time investment throughout companies of all sizes. Today, the majority of businesses are focused on core operations, and only a small periphery is spent on growth and innovative initiatives. I strongly believe that generative AI has the potential to alter resource investment in growth and innovation at a scale that we haven’t experienced in the past.

This could be very different from many other technologies that came before. It has a much larger impact on every aspect of a company’s operations and consumer functions. I also think that it will free up time and resources through automation and collaboration tools.

We also know that consumers are being rapidly trained with AI-powered experiences and the expectations are going to climb very soon. In a 3 to 5-year period, I do see a paradigm shift in time, resources, and mindshare deployed to emphasize growth, innovation, and newness powered by AGI.

Extra question: You shared an image with us. Could you explain the key points? I know you’re talking about this growth mindset of AGI really transforming how we think about innovation.

I see a business as two concentric circles. One is core operations and on the periphery is growth, innovation, and newness. This is backed by the data with companies traditionally growing around 6% – 10%. Only some startups have meteoric growth.

However, with the deployment of AGI, the core operations inner circle is going to shrink. Mundane tasks will be reduced and growth and innovation will increase at a very significant scale, because now the right tools will be available for it.

Question 3: Early Learnings: You make digital transformation the first stepping stone to true business transformation. And AGI is the catalyst, the enabler of that. While at William Sonoma you deployed some AGI initiatives. Do you have any early learnings?

First, I think that for every layer of technology that comes in you need a team, and you need yourself to be aware and have learned and gone through the process of building some expertise for all the previous technologies. So it’s microservices, platform data, big data, AI in general, machine learning…

Then, AGI comes as an additional layer on top of all that. You have to start by establishing a test-and-learn model with a small team approach – hopefully, if you’re reading this, your company has already started.

I also think it’s key to establish and prioritize relevant and valuable use cases, and to understand where generative AI could help. These use cases need to be quickly paired with LLM solutions vs. people just talking about what could happen. Not everything will work out, and that’s okay. You need to look at what is promising and then set a test-and-learn iterative approach to shape the product or feature your teams are working on.

Extra Question: What is ready from an AI perspective (the models and LLMs available)? And what are your relevant use cases?

For retail in general, it has been content generation and product descriptions. They have proven to be very, very productive. A chatbot helps not only by just checking your order but also some product expertise can be outsourced to that. Designing the look, shoppable images and videos, buying the look, buying the experience, searching, and recommendations improvements by applying AGI and LLM models, have all been very important aspects of the work. Also, 3D and photo-realistic imagery is becoming very common now.

On the other side, supply chain and operational efficiencies are being improved upon by predicting and optimizing shipping delivery routes (which used to be a very transactional table in the past). Now you can dynamically learn what is the best route for delivery. You can also do some returns, prediction, and accuracy, which will help you throughout your entire supply chain.

You could boost associate productivity through code co-pilots, which we tried, and collaboration tools to optimize communication. There are companies now that are talking about inside retrieval from all the documentation that’s available, and providing you with suggested actions that you can pick and choose from.

Talent acquisition and talent management have also come up as a new topic where AGI is being deployed.

Question 4: Metrics: I presume there are a bunch of use cases that you had to challenge inside of the organization. How do you rationalize the list of use cases? Did you apply some business metrics to them? How did you prioritize?

I think the easiest thing to pick is where you already have a pain point in the business. Then you try to look at it from both angles, as I said.

One is defining the problem. I think we should always stop and start with the pain removal, before diving into growth.

As a technologist, your job is to figure out what technology is ready. For example, what kind of element models are ready for this deployment versus others? You can spend cycles and months and months, and you find that the lift was only 2% which you could have done by changing the color of the button on a screen, for example, or a location.

So the metrics are important to your point, and I believe that generative website experiences are going to be an area that’s heavily impacted. So I envision that the typical search bar on the website is likely going to go away. It will be more of a generative website experience where you’ll find more relevant and personalized results. So search, recommendations, and chatbot are all going to be meshed into a more generative website experience based on the time of the day, your original location, if you are traveling, or if you’re on an iPad versus a phone.

It’s about giving the customers something very relevant in order to lift the conversion through the roof. That’s the current opportunity for AGI.

Then, the entire commerce funnel has clear KPIs at every step already established. For example, conversion at every step of the funnel: AOV, AUR, category penetration, shipping costs, etc.

Still, while deploying generative AI at each of these commerce steps, we can’t only measure the KPIs as we used to. In the past, you would usually deploy something and achieve a 3-4% conversion lift, or, revenue-per-visitor for that particular small step.

I believe the power of AGI to validate is that its impacts have to be at a much larger scale and pace. For something you were trying to do over 6 months, the way I would vet an AGI solution, in the beginning, would be: Are you shrinking my cycle from 6 months to 2 months? Or one month? Or are you talking about a 5% lift and you’re bringing an additional newness to my solution and experience that could give me anywhere from like 20 to 50% growth across that segment?

I think that’s the way you look at metrics here. And if you can even reach the halfway mark, I still think that’s a fantastic success vs. still trying to just get a 5% lift in conversion. Avoiding bias and cannibalism of other KPIs over a period of time is an important thing to watch as well.

Another important category in my mind is the analytics to track the AGI progress. I think there needs to be a shift from existing transactional reporting to more predictive trend analysis, facilitating better planning and decision-making.

Question 5: The ‘Buy versus Build’ concept: There’s a technology decision as well. You are the technology leader and in all of this effort at some level, you have to decide: Do you go with existing vendors? Do you take on some of these new vendors? Do you build your own model? What’s the ‘buy versus build’ thesis that you believe we should be thinking about? How do you go about making all of those decisions when it’s an early adopter market?

I practice a balance of this because you can’t be just a buyer person–you don’t need to be a technologist to be a buyer. However, if you’re just building, then you get too much into the weeds of proving yourself and your teams.

Everyone is new in this space because it’s all new. However, I believe that over 50% of the enterprises that have built LLMs from scratch are likely going to abandon their efforts due to cost, complexity, and technical debt. It’s not going to be manageable.

So these foundational ecosystems, like OpenAI or Azure providing a view into OpenAI, or some niche companies who have built on top of OpenAI, are credible. These provide you with a safe playground. They also provide you with learnings from their own data, and models at an unprecedented scale. You can’t compete with the data Google has, or Azure brings, etc.

That being said, you could deploy all your inference models and orchestration LLM models on top of their models to add even more value and specialized results for your business and problem space.

That’s the build versus buy combination.

You’ve got to watch it. But at the same time, I believe that if you don’t build muscle internally, then at some point you’ll just be running blind. You won’t know what you’re doing and you’ll be trapped working with all the major brands. So it’s also important that there are groups of people within the company who continue to evaluate and approach the latest and greatest open models for relevant business cases in order to validate the offerings from the large ecosystems and vendors. This ensures that you’re not missing out on something difficult on account of suffering from vendor lock-in.

Question 6: Gaps and Issues: What do you think some of the gaps, issues, or impediments are for large organizations? I know this isn’t a technology-only issue. What are some of the issues, or maybe even gaps in technology, that you believe need to be addressed?

I think the overall core talent pool is very small, and by this I mean people who know the guts of data and the guts of AGI. Tools like ChatGPT make it seem easy. The talent that says “I can get started by generating great prompts” is plenty. You don’t need to be a technologist to do that. But the core talent, to my previous point, is still very small.

However, for any large-scale enterprise, it’s critical to have some core talent that understands the guts of AI, the anatomy of these LLM models, and, more importantly, the criticality of data preparation.

So I think companies should also invest in their existing deep experts in data and platforms and get them motivated and trained on AGI. That will help integrate them with a new generation of LLM engineers. So the total product as a team will be more holistic, versus just rushing towards building LLM models.

To me, this remains the biggest challenge for most companies. And this despite all of the investments in the past decades that I’ve seen in data lakes, and data oceans or pools, whatever they’ve created. Still, your results and LLM insights are going to depend on your data. So preparing that data and the quality of that data is going to be very important.

Question 7: Responsible AI: We talked about bias and hallucination, trust, and risk governance issues. What do you think about responsible AI? And what message would you give to your senior leadership on how they should think about it?

From a business perspective, the scale of impact of AGI has to be much larger than the normal projects you would do in tech investments. And that’s the promise it should bring. Therefore, it must be vetted.

For that reason, I don’t even like to call it artificial intelligence. I’d rather call it Accelerated Intelligence, because that’s what it should deliver. If it’s not giving you accelerated intelligence to make an impact, then why even bother building all of these models?

AI should have constructive business goals and measurable KPIs and transparency. If you can, up front, define that this is the part that an LLM model is going to produce, and you can measure and provide that transparency to your business partners and leaders, I think they will come along for the journey. They will want to get trained and educated. They can even bring their own insights, and challenge you with the right insightful questions about the KPIs and measurements.

Don’t treat it like a black box: “We’re doing some research, and we’ll tell you where we find a nugget.” Rather, be upfront and defined. If there are seven failures, it’s good for the company to know that it’s not mature yet, even though our teams have tried seven times. Or you want to be able to say: “In this particular domain, it is mature, and it is giving us results, and the results are realistic.”

When it comes to the responsibility of AI, there are other aspects to consider as well. It needs to be protected from biases and corrupted training data or user feedback, which is called data poisoning or prompt injection attacks.

So it’s important to transparently define both of these issues in order to ensure correct outcomes across the various stages of development and deployment. Large vendors and ecosystems will need to focus here as well.

Another important aspect is that generative AI and its proliferation poses a big risk to organizations as it provides a significantly larger attack surface for cybersecurity issues.

It also provides a new attack vector, where it’s much harder to identify anomalies. So to me, data should be protected all the time and for any new deployment. One of the types or ways to approach this is that we should implement a safe test-to-learn approach with only necessary data provisioned in a sandbox environment. The solution runs to understand the scale of the idea and the impact and validate its safety before we roll it down to the full dataset of the organization, domain, or department.

Another idea I have is that there should be guided AI-type platforms like OpenAI. I believe there is an overarching need for these platforms given it has such a global impact. In parallel to OpenAI, this guided AI platform can govern all AGI implementations in an enterprise, map them each to their planned outcome, and then also catch signals proactively.

Even 100 implementations may cause long-term deviation from company goals.

If you implement AGI across many different departments, individual teams may be doing fine, but not aligning with top-level initiatives. You might have the goal of driving operating margins in a particular direction, and two individual teams could be polluting everything, even if the rest of the solutions are working fine.

There needs to be a parallel platform that can provide this guidance watchtower aspect, and that is not specific to any specific company. It’s a larger platform that could be deployed in enterprises and would watch out for every level of possible anomalies and deviation from a specific implementation of an LLM, but also tie it back to overall company goals, and see if any of the LLM implementations by themselves, or combined, may muddy the waters.

Yasir Anwar is an innovation enthusiast who is an Entrepreneur at heart and a Technologist at execution. #Techpreneur

Yasir is on a D.I.E.T – Disruption. Innovation.Execution.Technology excellence.

Yasir is shaping the strategic technical direction of the companies and transforming technology for the next generation of retail – the world of connected commerce. Yasir has led end-to-end business and digital transformation for companies through technology excellence, innovation, and digital prowess focused on customer experience.

Most recently he was the Chief Digital and Technology Officer for Williams Sonoma Inc. leading all digital, technology, and IT functions. Prior to this, he was the first CTO for Macy’s, where he drove the growth of e-commerce channels, and Omni capabilities and led the merger of the previous digital and CIO organizations under one mission and culture.

Yasir is also a Lean practitioner at scale and established the LeanLabs culture at Macy’s which grew from one LeanLabs team to 40+ teams contributing significantly to revenue increase, cultural agility, and innovation. Before that Yasir had many leadership roles with WalmartLabs, SamsClub.com, Walmart.com, and Wells Fargo, and executed large-scale projects across various domains like manufacturing, ERP, Auto Industry, Learning Management Systems, etc.

“If people are not laughing at your goals, your goals are not big enough.”

Today we welcome Dan Elron, Managing Partner, Enabled Strategy, to The CXO Journey to the AI Future podcast.
Until recently, Dan Elron was Managing Director of Corporate Strategy, Technology, and Innovation at Accenture, where he helped drive strategy for their technology business. He has advised many key clients, working with CEOs, corporate boards, and leading academics and policymakers across the globe.

Most of his client work has been in the high-tech, telecom, and financial services industries, and more recently, the automotive industry, as it began to integrate advanced digital technologies. He was also the Information Technology Industry Advisor for the World Economic Forum. For the past decade, he has worked on the impact of artificial intelligence across large enterprises, and anticipates significant disruption during the coming decade.

Question 1: Current Job: Could you tell us a little bit about your background and how you ended up where you are today?

I’ve been in management consulting for a long time. I recently retired from Accenture after working for many years in the tech and telecom industries, as well as other large global enterprises. I eventually began working on Accenture’s own business and technology strategy, its ecosystem and its relationship with both large providers of technology as well as startups.

Now I’m working with startups as a mentor, as an investor and advisor to investors, and with large organizations that are struggling to integrate the amazing wave of technology that’s coming to us, including the topic today, which is Generative AI. But as you can imagine, that’s just the tip of the iceberg. Many other things are happening over the next 10 years, I think not just in IT, but in IT-enabled technologies, including in healthcare, biology, agriculture, etc.

So there’s a lot of interest today around leveraging technology for strategic value, and I really enjoy working with clients on that topic.

Question 2: Generative AI: You’ve had so many conversations at a strategic level with thought leaders and major corporate teams. Is there something unique about Generative AI or AI, specifically, that you think means higher priority from a leadership standpoint? How much priority do you think we should be putting on this?

I’m reminded of Yuval Harari, who wrote the book Sapiens. There’s something he said that really struck me; he said that using Gen AI is “hacking the operating system of humans.”

Language is the operating system of humans, and if we’re hacking that, even if it’s primitive hacking today, we’re impacting a lot of the information flow among humans, a lot of the transactions between humans, and eventually a lot of the emotions between humans. Ultimately, all organizations are made up of humans, and they don’t work very well if the humans don’t work well with each other and communicate with each other.

So here we are, introducing a new agent. An automated agent that speaks our language, generates our language, and can help or hinder how an organization works. So because of that, I think it’s extremely important. I don’t think anybody understands exactly where it’s going to go. We’re early in the technology’s development; every month brings something new. And we’re fascinated by what’s going on today. But it behooves us to think about the likely evolution over the next 5 or 10 years, which is very difficult to predict. It will likely extend from language to images, and from the digital to the physical world. But it is fundamentally disruptive.

So the answer to your question is, yes, I think it’s extremely important. It doesn’t necessarily mean that this is where you need to spend the most money as an enterprise, but it probably means that this is where you need to spend the most thinking as an enterprise.

Question 3: Early Learnings: How have you executed a Gen AI strategy for your team or organization and what initial learnings or initiatives can you share?

I was shocked by the early use cases even a year ago. Very basic things such as feeding support tickets into a model, reducing the speed of figuring out those problems, and reducing the level of experience required to address those problems was truly astonishing.

So very impressive gains came very quickly. And not necessarily in the areas that we predicted. That requires experimentation which I think most folks are doing right now, and it requires creativity. There are a lot of underappreciated sources of knowledge that you can feed into Gen AI.

And it’s important to note that I make a distinction between Gen AI and the rest of AI, including machine learning. I think these things integrate with each other and need to work together; we can’t just focus only on Gen AI. But when you do that you get amazing results. Not necessarily always the most strategic results in terms of market positioning, but certainly impactful in terms of speed, cost, and as I said, the level of talent required to execute a process.

So I think the early learnings here are that it can be transformational for many processes. It requires experimentation. It requires integration, as I said, with other technologies, whether it’s AI specifically, machine learning and associated technologies, or with other technologies. If you’re in biotech it means integrating with the latest learnings in science and biotech. And that’s something that we’re beginning to see.

There’s an amazing difference between the few leaders and companies that have been at this for a year or six months, and those that are just getting started today. So if you’re not doing a lot of that, you’re already behind.

The other lesson learned, and something that concerns me, is that a lot of companies when they think about Gen AI, think: “What should I do? Which department is it? Customer care, chatbots?” And they rarely think about the ecosystem as a whole.

The suppliers are going to use the same thing, and you can work much better with your suppliers. Or sometimes be disintermediated by your suppliers if you don’t think about how they’re going to use Gen AI. And the same thing will happen with other partners: supply chain partners, government partners, your customers, etc.

Many of today’s use cases I see are internally focused. We need to start looking outside. Odds are, somebody outside our ecosystem will have better ideas than we do, statistically, in terms of what to do with Gen AI.

In some cases, if leadership teams aren’t careful, companies will get disrupted. Because somebody will build a platform, and products and services will be consumed by some kind of automated agent that they design and control. And companies will have to respond.

So the early concern I have here is: you really need a holistic point of view. Everyone is looking at their competitors today, but consider the whole value chain, how you’re delivering value to customers. Who has more power? Whose power is going to be amplified by the use of Gen AI? And, unfortunately, whose power is going to be diminished because they still have too much friction? For example, we’re making information more easily available. So the competitive dynamics in many value chains are going to be disrupted by that. It’s just beginning now, and it’s going to accelerate.

Because of this the role of the Chief Strategy Officer, which maybe wasn’t that significant five years ago, is really becoming much more significant.

Question 4: Metrics: So how do you move a team to begin? You mentioned the Chief Strategy Officer. How do you create metrics that set an agenda? How will you prioritize where to deploy Gen AI solutions? What areas or use cases are at the top of the list?

Before we go there, I think the strategy to implement Gen AI and AI in general, is potentially strategic. But it’s also so important tactically, in terms of cost reduction, and process design.

The strategy has to be driven both top-down and bottom-up.

The experimentation that you and many others talk about needs to happen and often should happen bottom-up. I don’t like the idea of having the Chief AI Officer; I know that’s a little controversial. I think everybody should be a Chief AI Officer. Certainly, everyone who reports to the CEO should be the Chief AI Officer for their domain. It’s not something you can delegate. It’s the same thing that I think about innovation more broadly, by the way.

But in terms of the impacts, they’re going to be different. Therefore the metrics are going to be different between the tactical bottoms-up work and the more strategic work. So clearly, for the tactical bottom-up work, the classic measures of productivity can be extremely significant, and indeed we are seeing sometimes major productivity gains already. Certainly, we see that in software development, and also in customer service, in repair, trouble ticketing, etc., they’re very significant. I would focus less on the ROI. One reason is that the cost side of geAI is still unclear, I think.

Prices are going to come down eventually, and I wouldn’t hamper the technology by allocating too much cost to it yet. When that scales, maybe, but for now I think you want to experiment and introduce it as quickly as possible and not apply the usual business case logic which slows everything down in many, many companies. I know that’s easy to say and CFOs don’t like it. But this technology is still early and I think that’s the right thing to do.

Also, for all executives, while there’s a lot of value in taking a two-hour course, if you’re an executive affected by AI or Gen AI, there’s a lot more value in actually seeing it across your organization, working, and hopefully working well – so you need to get close to the work in the field and not treat it as a standard ROI-based effort.
In terms of the metrics, I think speed is extremely important. Because, as we know, speed is strategically valuable in just about any context.

Another thing to consider is the business process: how many steps in the business process are automated versus manual? How can that change? It’s a very basic thing. This is something we looked at in the past when we did reengineering for those of you who remember that – efficiency and effectiveness.. But that will help you to understand the ability of your organization to change by applying GenAI, agents, etc..

Also, how susceptible are you to disruption? If something that took many steps suddenly becomes much more simple with this technology, it could be that your competitors, or new entrants, can do that with a new platform and overcome the complexity that kept you successful as an incumbent.
So some of these metrics are not traditional, but because we’re early in the technology, we want to understand the strategic value.

Another set of impacts that I find very interesting addresses the work experience and training required. We all know there’s a shortage of talent and that Gen Z has different priorities and different requirements for their jobs. So what is the impact in terms of the experience, training, and job satisfaction of introducing this technology? Therefore, what does that mean in terms of your labor demand, the kind of labor, and the amount of labor that you’re going to need over the next few years? How do you design the new AI-enabled roles while maintaining what humans value?

To give an example of impact, one case I know of is in the automotive industry. The average experience required to solve a particular problem went down from the need to use an engineer with 5 to 7 years of experience to engineers with 1 or 2 years of experience because they were enabled by Gen AI. That’s extremely powerful, and raises the question of, for example, how to keep both levels of engineers engaged and learning.

In markets such as Europe, where many engineers are retiring, the use of GenAI, including automating processes via genAI agents, will be extremely helpful. So, to your question, preserving knowledge and reducing labor demand are metrics that should both be considered.

And the last metric which I don’t really think you were asking about, is risk.

We tend to attribute too much risk in some cases to AI because of all we read. What is the real risk here? I encourage companies to evaluate the risks specifically for each use case.

What’s the worst that can happen? And how often do we have problems? Obviously, it’s still early days, we’ll have problems. But are they show-stoppers, can they result in serious, systemic injury or damage, or are they things that can be managed and fixed?

That perspective should help management teams overcome the fear that in some cases prevents them from doing anything. Even in a law firm, there are many, many things you can do with Gen AI that are not going to get you in trouble when you get to a judge who doesn’t like a brief that you’ve written with genAI.

And so it will help to have a solid, simple framework for saying, say, this is high-risk, high-reward, or maybe high-reward, low-risk. That helps you with the deployment strategy and helps you measure whether or not you have a balanced portfolio of tech applications in terms of risk and reward, and to be prepared for clean-sheet disruptors who will start with genAI as a basis for designing their processes.

Extra: You came from working at a management consulting firm for years. How do you think management consulting will evolve through this?

I think because the industry is populated by people, who, as an economist called them, are ‘symbolic manipulators’ (people who work with symbols, whether the symbols are software or words on the Powerpoint), that the industry is going to be perhaps one of the most affected by this ‘hacking of language.’ And if the language is structured, such as computer code, the disruption will be much more serious.

There are still barriers to this disruption. It turns out that, you know, PowerPoint, for example, is not the best way to train a large language model. But we’re going to get there.

From my perspective (and this is an industry I spent pretty much all my career in), the industry needs to reinvent itself. The impacts on software are definitely already happening. I just saw various projects where companies are using software to develop complex systems or at least pieces of complex systems that would have cost tens of millions of dollars and now can be done much faster with less experienced talent. So it’s disruptive there.

It’s also disruptive in something that the industry does, which is to take information, aggregate it, and deliver it in a customized way to a client.

It’s an industry that really needs to rethink what it’s doing. There are huge opportunities for the industry in redesigning the ecosystems, for example, for the use of agents; in rethinking how work is organized; in enabling the human transition; in making sure the benefits are captured and risks managed. The consulting industry did that in the process reengineering days; it is time to do that again and be willing to let go of work that is now highly automatable, however hard this is.. When a new technology is really threatening, sometimes you don’t move very fast, or you focus on small projects and proofs of concept, and I think that’s very dangerous for the industry. Technology providers would love to capture the value that used to go to consultants and integrators during past tech transitions. This time it could happen much more quickly, before the industry has managed to change how it works.

Question 5: Gaps and Issues: What do you think are some of the gaps, issues, or impediments for large organizations? I know this isn’t a technology-only issue. What are some of the issues, or maybe even gaps in technology, that you believe need to be addressed?

Well, I’m going to discuss it with you as a venture capital firm. There’s a terminology we haven’t used for a while, but maybe you’ve heard of it, it’s called the ‘- ilities.’

These are attributes of technology, especially computer technology, which include scalability, manageability, integrability, security, etc. Those are often missing for Gen AI – I think we all know that. But I don’t think there’s enough funding that’s going towards these things, which may not be that sexy, but for an enterprise to use this software in a reliable, predictable way, we need a lot of investment in those.

Another gap that kind of addresses something that I mentioned earlier is risk mitigation and risk management. We know that technology is far from perfect (think: hallucinations), but there are very few tools at this point to tell us what is dangerous and what is not dangerous. Or: to help enterprises figure out where to put the technology and let it run, where to turn up the temperature on a model, which means where to let it be more creative vs. use a completely explainable model. Not everything has to be explainable, that would be terrible if we required every model to be explainable. I know a lot of companies think they should. I don’t agree with that.

There are places where you can take risks, where creativity is good, and where, frankly, you don’t care whether you know how the answer was received. You probably can tell that the answer was a good one and the risk of not explaining how you got there is acceptable. I already see how big companies are unnecessarily preventing those uses.
We are also missing good approaches to training. We know that even an hour of training on using language models has a huge impact on productivity. But what about a day of training? What about a month of training with mentors? What kind of training? We still don’t know.

Another gap is the change management. People in many organizations, up to the very top are, in many cases, frozen. They will say, “Yeah, let’s have a few pilots here and there. Let’s reconvene six months from now, and see how these pilots have gone.” In some cases it’s fine, but in many cases, it’s too slow. So what kind of change management can we implement in order to encourage departments and product groups that are susceptible to disruption, or which have an opportunity to really do much better and generate a lot more revenue? How do we encourage them to be comfortable with the technology and to move from exploration into production? One lesson seems to be that it is better to deploy the technology to teams than just to individuals.

Even example case studies, videos, and interviews/podcasts such as the ones you’re doing in this series can give confidence to organizations to move faster. And right now I think many are saying they’re moving fast, but when you really look at the rate of implementation and how close they are to getting something into at-scale production, the answer is often not very good.
You extrapolate the things that they think could get into production and you see that they’re going to be a year or two years away, which for the more basic things, or, say in the IT shop, is way too long.

Extra: You’ve helped people get through this change management obstacle many times in your nearly 30-year career at Accenture. How did you get people fired up?

Well, it is challenging. But yes, I remember even having to convince many telecom companies that broadband was a good idea. And here we are with something that probably is much more disruptive than broadband.

Fear and greed are very helpful here.

For example, if you paint the picture of an ecosystem that’s full of agents that decide who should buy what, at what price, and from whom and then look at your own process and see how many steps and how bureaucratic what you’re doing today is, this creates a degree of fear because your competition could adopt these technologies and be much better, faster, and potentially cheaper.

The more I read about these competitors to Amazon such as the Chinese retail and e-commerce players who are coming into the US, and how they’re using not necessarily Gen AI, but AI more broadly, or technology in general, and how quickly they’re moving, that should strike fear into all retailers and other companies as well, maybe even highly regulated, often inefficient, protected industries such as insurance, who may intellectually understand the situation but don’t move very fast.

About greed, I think we all know how that works. It can be very powerful with the C-suite.

The cost reductions, as we mentioned, the reduction in the talent requirements for addressing business processes, the simplification of organizations, the integration of agents, not just today’s Gen AI, but what’s coming next, etc.: when you describe those and provide real examples in a C-suite you see eyes light up. They understand that this will have a bottom-line impact on their organizations. The revenue side may be more challenging, but the examples are coming quickly here as well- for example in simple upsell situations.

The biggest concern I have around getting people fired up is that there’s so much negativity about AI right now, especially in Europe. The concerns about the risks, the amount of regulation, and the issues around hallucinations that could get companies into trouble are all valid. But I believe that’s putting many speed bumps in front of the exploration and employment of the technology when they’re not necessary – or when it is others, such as regulators, or your suppliers, who have to take the lead. You may not be able to wait for these often slow processes for what you are doing inside your business, despite the negativity.

We often don’t like where we ended up with the Internet, and as a society we should learn from that and address these issues quite forcibly, but we should not let the sins of the past prevent us from dealing with the opportunities and challenges of this new set of technologies.

So these are some of the challenges we need to overcome. These are early days, and competitive advantage often is created in the early days. New models are created by innovators; in some cases, being second or third doesn’t work, especially when you look at platforms and other business models, new ecosystems, etc. Every industry is different, but I think having a view of what might happen, having scenarios, and moving quickly when necessary, makes a lot of sense.

Question 6: What would you say to a C-level executive on how they should be thinking about going forward?

My sense in the last few months, having been in this business for a long time, is that I have never seen a technology that’s moving as quickly with the potential impact that Gen AI has. This is partially because it’s built on top of the internet and a lot of prior investments. But we’re now accelerating, and I struggle to imagine what and how quickly things can happen.

So that’s the message I would leave. It requires making efforts to learn and reflect, to really think about the medium/long term impacts and how your business and ecosystem could change, including how you make money. As I mentioned, there are lots of lessons from early adopters, from China, other industries and places.

The second thing is, don’t delegate. As I said, hopefully you and your team can be your own Chief AI officers, and you don’t need to hire one. This is something that every senior business executive needs to take seriously and think about, at least for their domain, and how that’s going to accelerate as we get into, for example, autonomous agents, as we discussed,
Yes, the change can be concerning and could well be disruptive, but I think the impact of technology on things we care about such as healthcare, the environment, or transportation is going to be dramatic over the next ten years, and hopefully very positive.

I am hopeful our children are going to benefit from this. We need to be careful, but also really be inquisitive and excited about what might come.

Until recently, Dan Elron was the Managing Director of Corporate Strategy, Technology, and Innovation at Accenture, where he helped define its technology vision and drive its strategy for its technology business, and where he advised key clients, working with CEOs, corporate boards, and leading academics and policymakers across the globe. He also helped lead Accenture’s Innovation Network and its relationships with large and emerging technology partners. Most of his client work has been in the high-tech, telecom, and financial services industries, as well as the automotive industry as it needed to integrate advanced digital technologies. He was also the Information Technology Industry Advisor for the World Economic Forum. For the past decade, he has worked on the impact of artificial intelligence on large enterprises, anticipating significant disruption during the coming decade.

Currently, Dan works on strategic topics and advises senior leaders at large US technology players, and mentors and invests in startups in the US and Europe. He teaches at the University of Virginia, where he is associate director of the Center for the Management of Information Technology, and also at INSEAD in France.

Today we welcome Elvis Cernjul, Chief Information Officer at the Ubique Group, to The CXO Journey to the AI Future podcast

Elvis is a distinguished leader in the field of information technology and operational excellence, combining a rich academic background with extensive professional experience. He holds an M.Sc. in Technology Leadership from Brown University (as of May 2024), an MBA, and a B.Sc. in Information Technology. Elvis has excelled in multiple high-level roles including Chief Information Officer, Chief Operations Officer, and Chief Information Security Officer. His career is highlighted by his skill in transforming IT infrastructures, pioneering digital marketing strategies, and leading teams in high-stakes environments. He is currently making impactful strides at The Ubique Group while continuing to drive innovation and operational excellence across the tech industry.

Question 1: First Job: Could you talk a little bit about your background and how you got to the position where you are now?

So, the Ubique Group is a collection of brands that source and sell commercial-grade office and home products through channel partners like Wayfair, Walmart, Amazon, and others, along with B2B and D2C models. Prior to my tenure here at Ubique Group, I served in a variety of roles including a stint as an Army Ranger, believe it or not, and eventually wound up as both a CIO and COO.

My background includes about 35 years of leadership experience. 25 of those are within technology with a particular focus on the retail space.

Question 2: Generative AI: Everybody’s talking about it. You’re an operator of a large organization and you probably have other things to worry about, but we in the investment world think it’s all the rage, of course, and arguably it could be. What do you think about it from a priority standpoint?

I think that for our business, and the retail space as a whole, it’s a top priority. It provides such a leap in capabilities that if it’s not embraced in some capacity you’ll fall behind your peers.

It needs to be a part of every conversation that involves the future state of the business. My roadmap here at the Ubique Group revolves around leveraging the AI capabilities of any given platform. It’s not even really a competitive advantage anymore, it’s quickly transforming into a commodity.

And as we all know, machine learning and AI have been around for a while, but ChatGPT, Copilot, and some others have ushered in the accessibility of large AI language models.

Question 3: Early learnings. I presume you’ve been experimenting with it and done some personal testing with ChatGPT. I’m sure some of your teams are using it. I’m sure bosses or business units are coming to you with ideas. So what are some of the early learnings about it?

We’re thoughtfully, cautiously, yet excitedly approaching AI through ChatGPT. Today we’re subscribed to co-pilot and we’ve been kicking the tires on Einstein through Salesforce and a variety of other solutions.

A couple of initial learnings:

First, it’s really difficult to get a peek behind the curtains on how companies that are offering AI solutions plan to use our data, and what true safeguards are in place.

Second, the results today are not always accurate, detailed, or polished.

All that being said, we’re tackling a few initiatives where AI is at the forefront of discussions, including new customer care processes, call center systems, content and copy, business intelligence… and my development team is using it quite a bit

When managing change of this magnitude within an organization, especially with AI, it’s important to position it as a complimentary service and not a replacer. So I’ve been trying to balance it with that message as part of my language to the organization.

Question 4: New metrics: If you could break down the use cases you just described, how would you think about the metrics applied to each of them? You mentioned developer teams. Are you able to realize that impact? How are you thinking about metrics?

One thing that I’ve seen gain a lot of traction is time to market on development through AI-enhanced coding.

One of the intangible ones that we’re just now playing with is bubbling up business metrics through conversational AI. So it’s asking the data to reveal things that the business may not yet know to ask.

When ChatGPT released advanced data analytics, I started taking data from FRED BLS and some other public information, and throwing it into a spreadsheet along with Google Trends word. Then, I started asking ChatGPT to come up with correlations that I specifically wouldn’t even think to ask. I was amazed at the results that it was providing.

So, as a next step, I started showcasing this functionality to other key stakeholders within the business, and that led us to adopt a budget to pursue improving our insights and data analytics hub and expanding that to a data lake with some generative AI capabilities built into it.

Question 5: One of the questions that CIOs often ask is “buy versus build.” You mentioned some investments in current tech today, including Einstein and others. So the existing providers, I’m sure, are coming to you with their AI offerings. Does that give you an opportunity to buy versus build? Is there a combination of that? Or do you choose one or another?

We’re strictly focused on leveraging what our vendors offer. So we’re not staffed to create or train our language models. It’s not necessarily a differentiator for us. So yes, we’re closely partnering with our software providers and platforms to provide that means.

Bonus: Are they giving you an opportunity to convey your use cases, and design toward your use cases, so to speak? Is it evolving that quickly? Do you see a receptivity on that side?

Yes, for sure. With our partners at Salesforce along with the data lake that we just implemented, it’s a pretty close relationship.

Question 6: Gaps: Where are the gaps you’re seeing today? Undoubtedly you see some divots in the field, things you need to address. Is that a people issue? Is that a process issue? Or is that a technology issue?

I think it’s a little bit of all of the above. It’s training, knowing how to ask the AI the right questions, and how to identify errors. There’s also staffing. There’s no position, at least in our organization, dedicated to AI. But that is something I see changing throughout the industry within the next few years.

One of the key issues, as far as gaps go, is how to best wrap guardrails around AI so that users aren’t accidentally sharing sensitive information.

Bonus: And is that just educating the team on how to best use it?

It is, yes. We’ve had a few educational opportunities for employees, and we’ll continue down that path as well.

But I also struggle with how to wrap a technological guardrail around it so that we protect our assets, such as our customer data.

Question 7: Responsible AI: How does an organization like yours manage to be responsible when it comes to leveraging AI? How do you think about it? What actions should you take as a CIO?

Responsible AI…Defining it is an interesting endeavor that for me includes data privacy and security. Especially since now, we’re a global company, we have different regulations to adhere to. This includes our level of transparency with regard to using customer information. It also includes mitigating biases.

I believe everyone in retail wants to ensure that things like product recommendations, pricing, and marketing strategies are fair and equitable across all demographics. The same type of concerns extend to the internal consumption of data. We want to avoid a learning model that amplifies and doubles down on inaccurate results, which I’ve seen.

But I’m proud that we’re championing ESG within our organization. Part of our thoughts, or at least my thoughts, about wrapping responsibility around AI, is ensuring that we’re considering the use of energy-efficient AI technologies.

Elvis Cernjul is a distinguished leader in the field of technology and operational excellence, combining a rich academic background with extensive professional experience. Holding an M.Sc. in Technology Leadership from Brown University (expected May 2024), an MBA, and a B.Sc. in Information Technology, Elvis has excelled in multiple high-level roles such as Chief Information Officer, Chief Operations Officer, and Chief Information Security Officer. His career is highlighted by his skill in transforming IT infrastructures, pioneering digital marketing strategies, and leading teams in high-stakes environments.

Elvis’s unique leadership approach is shaped by his military background as a Combat Veteran and Ranger, bringing a blend of strategic discipline and resilience to his corporate endeavors. Currently making impactful strides at The Ubique Group, Elvis continues to drive innovation and operational excellence in the tech industry.

Today we welcome Naveen Zutshi, Chief Information Officer at Databricks, to The CXO Journey to the AI Future podcast. He joined Databricks in January 2022, and is responsible for their IT solutions, driving transformational programs to help the company scale its consumption-based business globally. He’s on the board of advisors for several fast-growing startups like Propelo & Torii, and more established companies including Zoom and Rubrik. He’s also an investor in many fast-growing startups and early-stage VCs.

Question 1: First Job: Could you talk a little bit about your background and how you got to the position where you are now?

I’m a computer engineer, so I love coding. However, I realized early on that maybe I was a slightly better manager. So I moved into management, mostly engineering management, for a while, but stayed technical, where I found my true calling.

In between, I’ve been at startups and in large companies. What’s been interesting is that sometimes you can take what you learned at high tech companies and leverage that in other environments such as retail. For example, I remember the work I did at Gap because it was such a huge opportunity. After all, there was a lot of legacy, and there was huge improvement to be made both on the e-commerce side as well as on the store side. In every job I’ve done, it has been amazing to work with so many smart people, which includes some of the incredible startup founders I’ve had a chance to partner with.

Question 2: Generative AI: Everybody’s talking about it. Maybe you can start by sharing your own view of how important CIOs should be prioritizing Generative AI. Is this a unique opportunity? We’ve all been through the mobile era, the social era, and the cloud era, Is this different? Could you put a little context around Generative AI?

I can tell you a little bit of my own story. Obviously, for me, the number is 22, because that’s when ChatGPT came out. I spent hours and hours asking all sorts of questions and getting some amazing answers. The initial hype was incredible, and I think, at least for me personally, I felt it was a seminal moment similar to, or maybe stronger than, any one moment that happened for mobile or the internet.

However, over the last year we’ve started to see what it can do, and right now there’s a disillusionment that we’re going through as an industry, but all the same I still feel incredibly confident that the future is bright for GenAI. In the meantime, there are a few thorny problems that we need to solve first.

Question 3: Early learnings. What are some of the early learnings when you think about Generative AI use cases there internally at Databricks? I’m sure you talk to CIOs in your role there. What are some of the early findings about how to become excellent?

The first use case that I’ve seen the most both internally as well as with our customers, has been the copilots, typically focused on software. We have implemented it for all our engineers and R&D.

We see a lot of our customers have rolled out different versions of it whether it’s Github, Copilot, Copilot X, or other versions of it. And they’ve seen various amounts of benefits, anywhere from 10-40%.

And I think there are other interesting use cases around software development. For example, test migration and test unit case creations, e.g. migration of codes from one language to another. That’s been a really good one that I’ve found for even applications like Salesforce. So, that’s starting to become a more established and more mature use case, and companies are mandating that every developer use Copilot in their daily work, which has become the new standard or the new expectation around productivity.

The second real use case I’ve been seeing is summarization. We want to drive summarization, and reduce hallucinations using RAG. And ultimately, we’re starting to see this action as intent-based. You know, where you take an intent, and have a multi-step process that actually achieves that intent. And I think that’s hopefully where the industry is moving. We’re seeing some early examples of that in the B2C space. But I’m assuming those will also translate into the B2B space.

Question 4: Gaps: So if that’s the future, where we have intent-based agents actually resulting in true productivity or workflow redesign, what are the risks in getting to that? What are some of the obstacles, issues, or maybe even gaps in the technology that you believe need to be addressed? I know this isn’t a technology-only issue. What are the headwinds overall?

Let’s start with RAG use cases: The biggest obstacle today is still reliability. You measure the reliability of answers coming back and in an enterprise setting your reliability needs to be pretty high, even close to perfect. And with GenAI use cases, you can reduce hallucinations, but it’s hard to achieve perfection. That’s one area.

The other area is unfortunately still the completeness of data. A lot of customers still talk about that. I don’t have all the data sets in a common form. For example, we work with a large air carrier. Do we have all of it in a data lake or a lake house paradigm like that large air carrier? Do we have the access control setup? So that’s another area. For instance, if I took out this unstructured data, and then ran it against a RAG model, but then you asked it a question, and I asked it a question, it should be privacy-preserving for you versus me.

And then, if you think about each summary, or each step of the process, if that reliability is very high, can I chain it together into a common model? Where can I drive content into this sequence of steps? For example, could I have it flawlessly plan an entire summer trip if I provide all the right parameters? Would it book the hotels, the activities, the flights, the meals?

I think that the validation step with humans in the loop is something that will still be required before it becomes truly autonomous. I think that we’re a ways from a truly autonomous state.

New metrics: We’ve watched the hype slow down as people don’t see the results they want. What are the necessary success metrics to prevent people from falling into that trough of disillusionment?

One thing I would say is to think about AI as a whole. What has been the success? The success has been dramatic, right? So there’s a dramatic success in revenue growth and profitability.

Let’s say you have so many use cases across AI overall that are now in production. And companies have seen revenue growth as well as profitability improvements as a result of AI. In fact, many companies are just AI-based as a result.

The same thing is going to happen with GenAI use cases too. I think the productivity benefit is the biggest one. If you look at the McKinsey report, it says 30% of all work could be automated by GenAI. I’m not sure if that is true or not yet. It’s a big number, but I would expect to see people start with some core use cases where you develop some level of success at scale.

Question 5: One of the questions that CIOs often ask is “buy versus build.” The idea here is that this is a new technology we need to learn. We’ve got people and staffing issues. But there are also existing vendors, and they’re offering AI-capable functions. Copilot, for example, as you mentioned. Do I do a little of both? Do I become an expert at building? What’s the “buy versus build” thesis that you believe we should be thinking about?

The way I think about this today is that there are two vectors: one is transitory, and the other one may be more permanent.

The first vector is thinking something like this: “Hey, if the field is still nascent, my SaaS vendors or my new startups can’t achieve scale yet, or they’re not working on the problem that I’m working on. Maybe I can experiment and build something in production. Maybe that will last for a year or two. By that time my SaaS vendors will catch up, and I will leverage their products.”

The second vector is that I have proprietary data, or have proprietary algorithms that I want to use against private data. And I don’t want to share this data with my SaaS vendor. I want to build it on a framework that is my own platform.

And I think there will be continued use of that, you know. That is Bloomberg’s use case. And it’s not just tech companies. I think non-technology companies will also have some use cases like that.

And I think on the buy side, what we’ve primarily seen is newer vendors coming into play whose GenAI capabilities we can leverage. And that has been very, very interesting. And so we’re using some buy as well and internally we’re using both buy and build.

So it’s a hybrid model and my expectation is that over time, with internal use cases, you will do more buy than build, but you will have some level of build done for sure.

Question 6: Responsible AI: This is a concept we’ve heard about in the press. There’s this idea around potential bias, hallucinations, privacy risk, as you noted before. How do you think about it? What advice would you give to a CIO?

I’ll mostly speak to B2B companies. I think B2C use cases may have additional privacy restrictions. So for me, customer data is sacred. How are we ensuring that the customer data is privacy-preserving? We’re talking to the customers before we use their data for training our examples or internal custom employee data. There is personalization, and there is privacy on that data as well. So you have this confidential and sensitive data. And in practice, you should have the right classifications on that data, so you just need to preserve those classifications back into your AI models as well. That’s one area.

The second area is policies. What kind of policies have you established? Make sure that you have really robust and clear policies established with legal and that you’ve educated your teams and your employees on those policies. There’s a lot of hype to use these tools, but you don’t want to inadvertently use them in the wrong manner.

Finally, having a good governance model in place is important as well. Depending on the use case, you may still have some hallucinations, and that could be acceptable, but in other use cases complete precision may be required. Understand what kind of use case you’re actually delivering. What kind of information do you want to create?

Bonus: Do you see an organizational role for responsible AI? And if so, on whose shoulders does it rest?

It’s hard for me to say right now. Today, legal, security, and IT are working together on this, but I’m assuming that in other companies, especially in B2C companies, that might become a specific role.

Naveen is currently the Chief Information Officer of Databricks. In this role he is responsible for Databricks’ information technology solutions, driving transformational programs to help the company scale its consumption-based business globally. Naveen is on the board of advisors for several fast-growing startups like Propelo & Torii, as well as established companies like Zoom and Rubrik. He’s also an investor in many high growth startups and early-stage VCs.

Naveen’s experience spans software development and infrastructure, from leading organizations from Fortune 500 companies to tech startups. Prior to Databricks, Naveen was CIO at Palo Alto Networks for the last six years, helping the company move to a cloud-based business, and integrated over 17 companies in that time period. Prior to Palo Alto Networks, Naveen worked as Senior Vice President at Gap Inc. where he was responsible for the company’s infrastructure, operations, and information security organizations. Before Gap, Naveen spent time at SaaS startups or high-tech companies like Cisco.

He earned a BE in Computer Engineering from Bangalore University and an MBA from the University of Arkansas.

Today we welcome Monica Khurana, CTO at Dodge & Cox, to our “CXO Journey to the AI Future” podcast. She joined Dodge & Cox in December 2017 as a visionary who has led the industry in creating products and solutions that are leading edge and sometimes the first of their kind in their respective spaces. Monica is a seasoned executive with over twenty-one years of management, operations, product, technology, financial planning, digital marketing, and cybersecurity experience.

Question 1: Current Job: Could tell us a little bit about your role? How long have you been at Dodge & Cox?

This is my seventh year. After a stint in technology, I moved into finance back in early 2000. I had to do a lot of data analytics, both in terms of looking at market data, as well as looking at client retention and client data. At this time, I built my first performance attribution system to evaluate how investment funds are doing and what was driving their performance. This was a huge success, and I was asked to manage hundreds of billions of dollars.

I was hired here to do something similar, which was to bring new capabilities from a technology perspective, including cloud capabilities, generative AI, and many different data and analytics solutions. The goal has always been to continue improving the performance of our funds.

Question 2: Generative AI: Everybody’s talking about it. You’ve been managing technology for a number of years, how do you see Gen AI as different? And how much priority do you think we should be putting on this?

I think we’re looking at an inflection point from a technology perspective. You and I have been around for a long time, we saw the internet, saw mobile, saw crypto. I think this is another inflection point.

However, generative AI isn’t new. It’s been around for a while, but ChatGPT made it so accessible to everyone that most of us, by experimenting and trying little projects to see what capabilities we have, saw that the promise of the journey is very real.

You no longer need to be a technological person to do a lot more than what you could do earlier. So we’ve been spending time considering: What value do we want to add? Where do we want to apply it? And, what projects are we working on today that could use it? Do you apply it to looking at investment data? Looking at Alpha generation? Looking at benchmark data and other market data providers?

The second bucket would be clients. Are we retaining our clients? Are we supporting them? Are we answering their questions quickly enough? We definitely need tools around that.

The third bucket is internal productivity. This could be pre-market commentary, code testing, or reviewing disclosures. There are many different ways we can get more productive.

Right now we’re mostly just dealing with where to start and how to prioritize all of these different applications.

Question 3: Metrics: Obviously, new technology always has to be justified. How will you measure ROI on this and choose one use case over another? What sort of metrics guidance would you give other CTOs?

Prioritizing all this is very complex. How do you invest and where do you deploy your resources? There are a lot of parameters you can look into, but a good one is savings over time. So, how do you quantify those savings?

Another good metric to consider is time. Where will you redeploy time that gets saved? Is it going towards strategy? Towards driving revenue? Towards product development? So it comes down to who’s going to get the value out of it.

The second piece is around the return on investment. What will it take to build a model? What will it take to maintain it, from the computing, research, and storage perspectives? And then, when do you expect a return on investment? One year, two years, three years?
There are certainly some low hanging fruits like chatbots and helpdesk. In those use cases, we expect a faster return. Things like product will take longer.

The third piece is going to be improving the employee experience. Ideally we want employees to be able to spend less time doing mundane tasks and more time doing creative, innovative, strategic things. The hope would be to see improvement in employee satisfaction and stakeholder satisfaction over time.

So it’s a complex mix, and we have to look at all of these together to figure out how to maximize the value.

Question 4: The ‘Buy versus Build’ concept: You talked a little bit about the effort of cost analysis, because with AI (and any new technology) you’re having to both teach and learn new products, new technologies, and new services. This could even include building new teams and resources. So is there a different level of quandary around buying technology that’s prepackaged versus building your own because you have custom requirements? What’s the ‘Buy versus Build’ thesis that you believe we should be thinking about?

I think this is a complex problem. All our vendors and key strategic partners are investing in the space. And we’re trying to see where they’re going and what they’ll be able to offer.

The second component is around data. They’re all building these capabilities on the data they own and the data they have access to. So we are seeing market data providers. We’re seeing auto management systems, all of them building in different capacities.

So our goal here is to see what our vendors are going to provide. And then the question turns to our own data: What do we do with that? Should we provide it to one of our strategic partners and have them build on it? Should we be applying these third-party LLMs and doing some analysis on our own?

Additionally, there’s this whole thing around hallucination, false outputs. We’ll also need to evaluate the output coming out of our vendors and strategic partners, not just our own AI resources. At the end of the day, we’re still held accountable to our shareholders and clients.

Question 5: What are some of the issues, or maybe even gaps in technology, that you believe need to be addressed? I know this isn’t a technology-only issue. What are the headwinds?

We’re already seeing a renewed focus on data. I mean, we’ve already had data governance, and data quality in the works for a bit. I think the importance of data has become more real now on account of generative AI. Understanding your data and having it structured in the right way so that these new AI tools can leverage it is important.

The second issue is around security. Risk of data loss is real. So how do you ensure that your contracts with vendors cover and protect your data?

And then there’s a risk of hallucination as I mentioned earlier. How can you really ensure accuracy around all this? Just like cybersecurity insurance, are we going to see more insurance providers providing protection around hallucinations? There are definitely some tea leaves swirling around that.

You made a good point about the human side of things. We’re going to see more AI operationalization meetings at the management layer and the leadership layer. Everybody needs to understand this. So what are the shifts in skills that are needed? Be it in terms of education, or be it in terms of bringing about some of these tools?

Getting started is also a bit tough. You want to start small, see some success, and then expand. So how do you bring that growth mindset into all this? There’s still a lot of figuring out to be done, and the challenges related to that.

And the last piece is regulation. The EU, Biden, as well as the India Digital Act, and the Canada Act. We’re still trying to figure out where that’s all going to land.

Question 6: Responsible AI: How do you think about it?

I think that we all have to focus on it significantly both from the perspective of the biases and the data it is being trained on. How much can you rely on the quality of your outputs when trying to apply it? Recruiting is a good example of this: you have to be so careful that no bias is introduced.

The second piece also is around looking at IP infringement. The verdict is still a little unclear on where some of this will go. I mean, is training on publicly paid available data a good thing? Not a good thing? And how do we make sure we’re not negatively impacted?
I think in 2024 we’re going to see a lot more clarity starting to emerge.

Bonus: What do you think about this market opportunity? You said it’s a high priority. But let’s say you’re standing in front of a large audience – your board, your leadership team – How would you tell them to think about the AI market right now?

At the end of the day, we’re at a big inflection point. AI has democratized IT across the workforce, and that is powerful by itself. The question now comes down to how we harness this power.

I think you and I talked in the past about what we would see in the infrastructure layer, the cloud layer, and the data layer. We should expect to see gen AI embedded into every aspect of everything we do, from the consumer side, to the investment side, to the data side. I can’t imagine a facet where we would not expect to see this.

The question is more around timing. How much to expect by 2024? And how much to expect by 2027? Where regulations are heading will be important too. AI is already important today, and I don’t think there’s any going back. It’s more about how far this will take us.

Monica Khurana has a diverse work experience spanning over several industries and roles. She is the Chief Technology Officer at Dodge & Cox and serves on the board Committee for T200 (a non-profit promoting women in technology).

Monica also held leadership positions at Guardian Life Insurance-RS Investments/Victory Capital Management, Cornerstone Research, MUFG, and MNM Partners Inc. She has been in Chief Information Officer and Chief Technology Officer roles since 2007, and has been responsible for various strategic initiatives, such as integrating acquired firms, transforming technology platforms, and aligning technology with business goals. Earlier in her career, she worked at Barclays Global Investors, CareCore/Varian Medical Systems, HP, and the University of Missouri-Columbia Hospital and R&D department, where she led projects in areas such as asset management, healthcare technology, and patient care systems.

Monica holds a master’s degree in computer science from the University of Missouri-Columbia, attained from 1996 to 1999. Monica also has a master’s degree in industrial engineering from the same institution, earned from 1996 to 1998. Prior to her postgraduate studies, Monica completed her Bachelor of Engineering in Industrial Engineering (gold medalist) from the National Institute of Technology (REC Jalandhar) between 1990 and 1994.