Gartner Analyst Shares the Practical Realities of AI Implementation

AI is reshaping industries with its potential to revolutionize workflows, boost creativity, and streamline operations. As organizations explore its transformative power, CIOs face the dual challenge of integrating this cutting-edge technology while addressing inherent risks. To delve deeper into the implications of generative AI, we spoke with Shubhangi Vashisth, Senior Principal Analyst at Gartner. She sheds light on how CIOs can harness this technology to drive value and innovation in their enterprises.

A significant challenge in AI deployment is that nearly 50% of AI projects fail to transition from proof of concept (POC) to production. This bottleneck arises from issues like data silos, lack of collaboration, and the incompatibility of machine learning models with enterprise systems. Many projects stall due to the lengthy processes involved in integrating AI models into production environments, re-coding them for compatibility, or iterating to maintain relevance with changing data.

“AI initiatives often stall because the journey from prototype to production requires a fundamental rethinking of workflows, technology stacks, and organizational priorities,” noted Vashisth. Moreover, the rapid pace of technological advancements demands continuous model updates and compatibility checks to remain relevant—a resource-intensive process many organizations underestimate.

She highlighted two concurrent AI races. One is the tech vendor race happening across the industry, but organizations should focus on the second race: achieving AI outcomes. Many CEOs still find it challenging to derive value from AI initiatives, as fewer than 50% of projects move from proof-of-concept to production. Each organization must define its own AI strategy, recognizing whether it needs to accelerate adoption or take a steady approach. It's crucial to align AI initiatives with business goals for tangible outcomes.

Excerpts from an interview: 

The Stumbling Block: From POC to Production

How should organizations decide on realistic AI ambitions?

First, it’s crucial to assess your organization’s AI maturity and goals. The approach will vary depending on the size of your business. For mid-sized organizations, large-scale AI implementations might not be realistic due to resource constraints. In this case, relying on packaged AI solutions or using AI embedded in existing software could be a practical choice. Departments like marketing or sales may also adopt their own specialized AI tools to meet specific needs.

In contrast, larger enterprises may have the resources and urgency to develop custom AI solutions in-house. They might build dedicated AI teams, acquire the necessary talent, and establish robust data governance structures. The key here is to first define your AI strategy and align it with organizational priorities.

It’s also essential to consider the outcomes you want to achieve with AI. Many businesses focus on productivity improvements, but AI can also be used to enhance customer experience, drive emotional intelligence insights, or even measure sentiment scores that help inform business decisions. Establishing clear goals at the outset will guide your AI investments and ensure alignment with broader business objectives.

What metrics should organizations focus?

To harness AI effectively, organizations need to first assess their maturity and specific objectives. Smaller businesses might lean toward packaged or embedded AI solutions within existing tools, while larger enterprises may prefer to develop custom solutions in-house with dedicated teams and structured governance. Defining clear AI goals—whether focusing on productivity, customer experience, or sentiment analysis—ensures alignment with overall business strategy and maximizes impact.

How can CIOs ensure AI projects deliver value and avoid common pitfalls?

Organizations must approach AI initiatives with caution, given that less than 50% of voice-of-customer (VOC) projects move to production. The primary challenge here is delivering real value. To overcome this, careful selection and prioritization of use cases are crucial. Additionally, managing costs is essential. A keynote highlighted that cost estimation for generative AI can be off by as much as 500-1,000%, due to hidden costs.

Continuous monitoring of expenses, akin to cloud computing, is vital. It's also important to track quantifiable metrics, not only business KPIs but also technology and risk KPIs, and potentially other outcomes, such as market differentiation.

Regarding AI budgets, some organizations might already allocate funds specifically for AI, but often the AI expenses are part of a broader IT budget. It’s essential to reassess the budgeting approach as AI becomes more integrated into the business.

What are some game-changing examples of AI adoption in industries?

In the pharma sector, generative AI has revolutionized drug discovery, reducing time and cost. In chemicals, AI is discovering new compounds and synthesizing protein sequences. Similarly, in consumer industries, AI has been used to develop innovative products, such as vegan dishes with unique flavor profiles. These advancements illustrate AI’s versatility across sectors.

How should CIOs approach AI benefits as a portfolio?

The "size of bets" concept is essential. Organizations must assess their risk appetite when pursuing differentiating AI use cases, as these often come with higher risk exposure. Each company has a unique balance between the value they aim to achieve and the risks they are willing to take. This is where the leadership team, particularly CIOs and C-suite executives, plays a critical role.

They must clearly define and communicate the organization’s AI ambitions and set expectations for both employees and stakeholders. By outlining the desired outcomes and the path to achieving them, leadership can guide the company through the AI journey, ensuring alignment and commitment across the organization.

With only 20% of CIOs focusing on mitigating AI's negative impact on employee well-being, how can this gap be addressed?

Start by identifying "deep productivity zones" within teams and selectively enabling them with AI.

To close this gap, it’s important to identify key productivity areas within the organization by evaluating employee experience levels and the complexity of their tasks. AI, especially generative AI, can be selectively introduced in these areas to enhance efficiency.

When there is human-machine interaction, traditional metrics like customer satisfaction or effort may not suffice. Instead, emotional intelligence and human-centered KPIs—such as emotion scores—become crucial. These metrics need input from experts in behavioral science, psychology, and ethics, fields that are increasingly playing a role in AI implementations.

What is the "tech sandwich" concept, and why are trust technologies essential?

In the "tech sandwich" concept, core technologies like trust, risk management, and security management play a critical role. These are often embedded in a wide range of applications, though some niche vendors specialize in this area. The focus on these technologies can vary based on the industry.

For example, regulated industries may emphasize governance, while security is a universal priority across sectors. The level of governance required can also differ between external and internal use cases—internal use cases might not need the same level of oversight as customer-facing ones. Therefore, organizations need to assess which technologies are essential within these three domains, d

How can CIOs make accurate cost projections for generative AI?

Accurate cost projections require:

• Careful use case selection and prioritization.
• Automated cost quantification processes.
• Defined success metrics to track progress.

Monitoring costs continuously is key, as generative AI often has hidden expenses that need foresight.

How do you foresee the role of CIOs evolving with AI becoming ubiquitous?

CIOs will play a central role in embedding AI into organizational workflows. From establishing Centers of Excellence to defining long-term AI roadmaps, their responsibilities will expand. They must also oversee governance, funding, and integration to ensure AI initiatives align with business strategies.

What advice would you give to CIOs on balancing AI ambitions with realistic outcomes and employee well-being?

Balancing AI ambitions with realistic outcomes requires a deep understanding of the technology. One of the first steps for CIOs is to ensure AI literacy across the organization. This involves educating executives and employees about AI’s potential and limitations. Many organizations develop specialized courses for senior leaders to help them stay informed about the latest advancements, while offering more targeted, practitioner-level courses for teams implementing AI.

It's also important for organizations to be realistic about the journey from proof-of-concept (PoC) to production. Transitioning a PoC into a fully functional product involves careful management of ambitions, costs, and technology. A more pragmatic approach could be to start with a minimum viable product (MVP) rather than jumping straight into full-scale implementation. Defining success metrics for the MVP helps set realistic expectations, enabling teams to iteratively improve the solution based on real-world feedback.

CIOs should prioritize employee well-being by managing expectations and avoiding AI fatigue. Implementing AI should be seen as a long-term process with measurable milestones. By setting achievable goals and aligning AI projects with the broader business strategy, organizations can ensure a smoother integration of AI while safeguarding their teams' mental health and productivity.