Unpacking the Realities of AI Strategy Execution

AI's Hidden Costs: The Enterprise Grapple with Shadow Deployments and Pipeline Chaos

AI’s transformative promise has captivated executive boards across every industry. Companies are chasing an "AI-first" mandate, pouring significant budgets into new tools and teams. But behind the hype, a less glamorous reality is taking shape: integrating AI into existing enterprise structures is proving to be a messy, complex endeavor, riddled with governance nightmares and unexpected engineering burdens. As we approach 2026, the discussion isn't just about *if* you'll use AI, but *how* you'll tame its unruly side. This is the core challenge Hema Raghavan, co-founder and head of engineering at Kumo.ai, and I recently dug into on the Stack Overflow Podcast. She's got over two decades in software, having previously spearheaded AI initiatives at LinkedIn, and she's seen the chaos firsthand. Her insights offer a stark look at the often-overlooked downsides of enterprise AI adoption, particularly the problems of "shadow AI" and "pipeline sprawl."

The Pervasive Threat of Shadow AI

The C-suite's enthusiasm for AI has naturally trickled down, incentivizing every department—from engineering to marketing and sales—to explore its capabilities. The obvious downside? This explosion of AI activity often bypasses the IT team’s control. Welcome to the era of "shadow AI." CXOs, CSOs, and CIOs are increasingly concerned about sensitive company data exiting the secure perimeter. Think about it: an employee might innocently use an unapproved large language model (LLM) service to "clean up a sales deck," inadvertently sending proprietary sales strategies or customer data into the public domain. Or perhaps they're granting an unvetted AI tool access to CRM data. Without proper governance, companies lack visibility into what information is flowing out of their systems, creating substantial privacy and security risks. It's a Wild West scenario for corporate data, and Open Claw, while exciting in its potential, only amplifies these organizational risks.

Strategies for Taming AI Governance

So, how are businesses fighting back against this data exodus? Raghavan points to a few emerging patterns. One effective strategy involves deploying AI models directly *within* approved platforms. She cites Snowflake's Snowpark Container Services as an example, where models operate inside the database or data warehouse perimeter, ensuring data never leaves a security-vetted environment. Kumo.ai itself adopts this model for its customers, ensuring their AI remains compliant. Other companies are opting for in-VPC (Virtual Private Cloud) deployments or establishing a centralized gateway. This gateway acts as a choke point, routing all external AI calls through a single monitored access point. This allows IT teams to scrutinize data ingress and egress. While implementing such a gateway demands significant resources, it provides a critical layer of oversight. Raghavan even suggests that AI itself can police these gateways, detecting sensitive data like PII before it leaves the perimeter. Another key control point lies in strictly managing *what* data AI systems can access, particularly in heavily regulated sectors like FinTech and healthcare, where access to electronic health records needs stringent, auditable controls.

Eliminating Pipeline Sprawl: Kumo.ai's Novel Approach

Beyond governance, the practicalities of maintaining complex AI systems are a massive headache. Raghavan shared a powerful anecdote from her time leading AI at LinkedIn. She oversaw dozens of models—everything from "People You May Know" to job recommendations—each fed by hundreds of data pipelines. One day, a single front-end tracking pipeline broke upstream, causing models to behave erratically. Debugging this "pipeline sprawl," tracing a fault through layers of interdependent systems maintained by thousands of engineers over years, was a "nightmare." Unmaintained pipelines and "bit rot" only compound the problem, draining engineering resources and introducing instability. This experience directly inspired the creation of Kumo.ai. (You can learn more about their approach at [http://kumo.ai](http://kumo.ai)). Raghavan and her team aimed to drastically simplify AI architecture by tackling the root cause of pipeline sprawl: feature engineering. Predictive AI models, like recommender systems or fraud detection, often rely on bespoke pipelines to aggregate data (e.g., "last 30 days of clicks") into features. Kumo.ai’s solution is bold: a single foundation model. "Can you imagine," she asks, "a company with one foundation model to maintain for all those use cases?" It sounds like madness, but it's remarkably elegant. The model uses in-context learning, much like generative AI, to query the database on the fly for specific use cases. For a job recommendation, for example, the model queries relevant user data in real-time, receiving context-specific examples to generate a response. This means maintaining one foundational model and one online service to handle dynamic database queries, essentially moving the "pipeline" logic closer to the AI model itself and drastically reducing the need for numerous, brittle ETL pipelines.

The Unified Data Warehouse: A Governance Anchor

This conversation naturally led to data architecture. Raghavan firmly advocates for a unified data warehouse layer, particularly for analytics and AI. A single warehouse, she argues, simplifies governance immensely. It provides a central catalog to determine which datasets models can access, and it centralizes monitoring of data flows. While acknowledging that differing performance characteristics can make full unification challenging for some companies—especially between online services and analytical databases—she sees a clear trend toward consolidating ETL processes into a single warehouse for AI applications. However, the online service layer presents a different challenge. These systems, demanding high performance for real-time recommendations on e-commerce sites, often can't rely on data warehouse performance. Online architectures also tend to be far more bespoke across companies, leading to fragmented, self-built governance solutions. It's easy to spin up a Postgres backend for a new application today, but this initial flexibility often comes at the cost of unified governance down the line, a problem that only truly surfaces when a company scales. Raghavan doesn't see companies building their own core databases anymore; that’s a commoditized space where external solutions are always more cost-effective. But certain components, like logging systems, can become choke points at scale, driving some large enterprises to consider in-house development when SaaS providers falter or costs skyrocket. In essence, the future of enterprise AI isn't just about developing powerful models. It's about architecting sane, secure, and maintainable systems that can handle the "messy truth" of real-world implementation. Companies like Kumo.ai are tackling these issues head-on, promising a more streamlined and secure path to AI integration. *(You can connect with Hema Raghavan on [LinkedIn](https://www.linkedin.com/in/hema-raghavan-1581075/) or reach her directly at [[email protected]](mailto:[email protected]).)*The rapid integration of AI into development pipelines is forcing a tough re-evaluation of fundamental engineering principles. What used to be straightforward architectural choices are now complex trade-offs, driven by the seductive promise of AI-driven velocity against the very real threat of technical debt and unmanageable sprawl. One of the most immediate points of friction, as Hema Raghavan, head of engineering at Kumo.ai, points out, is the database landscape.

The Sprawling Database Dilemma

We're past the days of a single, monolithic database serving all purposes. Today, application designs often splinter into specialized data stores: time series databases for logs, key-value stores for production, and, increasingly, vector databases for AI embeddings. The appeal of these specialized solutions is clear: optimized performance for specific data types and access patterns. But here's the thing: each new database adds another "source of truth" to maintain. Raghavan articulates the frustration shared by many engineering leaders, stating her first question for any new design is, "can we not have one more database?" The problem isn't just the sheer number; it's the constant synchronization. Vector databases, for instance, are particularly tricky. When the underlying AI model shifts, those embedding vectors can immediately go out of sync, creating a maintenance nightmare. The fewer of these distinct data sources a team has to manage, the better off they'll be. This isn't just about efficiency; it's about avoiding cascading failures and keeping systems reliable.

Navigating AI Experimentation and Technical Debt

The current AI space is a hotbed of experimentation, which, while exciting, also contributes to this architectural chaos. Teams are often split, with some going all-in on Retrieval Augmented Generation (RAG) coupled with vector databases, while others champion prompt engineering as the primary interface. This lack of standardization inevitably creates problems, making it incredibly difficult for engineering organizations to build consistent, maintainable systems. For CTOs and engineering leads like Raghavan, it’s a delicate balancing act. On one hand, you want your teams to experiment; you need to understand the possibilities. Yet, you also need a strategy to rein in that exploration once the initial phase is over. Organizations that have been at this for a couple of years are beginning to see what works and what doesn't. Some might find prompt engineering sufficient, while others discover real ROI from RAG. The answer, as always, is use-case specific. However, the longer-term concern is pipeline sprawl. Unchecked "vibe coded" solutions, disparate databases, and varied patterns accumulate, creating significant maintenance burdens. Raghavan insists on three core metrics as her team's "true north": velocity, P0 bugs per release, and Mean Time To Root Cause (MTTR). The danger with the current "agentic" and "vibe coding" trends is that teams often chase velocity at the expense of P0 bugs and MTTR, a trade-off that rarely pays off in the long run. The solution, it seems, lies in a renewed focus on governance through architecture. At Kumo.ai, they're starting to insert agent-stored MD files directly into repositories, essentially giving coding agents an "opinion" and guiding them to adhere to preferred design patterns. It's an interesting approach to codifying best practices right into the tooling itself.

The Evolving Role of Engineers in the AI Era

Perhaps the most profound impact of this shift is on the engineering workforce itself. The very definition of a "senior engineer" and a "junior engineer" is changing. Senior engineers are no longer just guiding their junior counterparts; they're also guiding AI agents. One senior engineer, Raghavan recounts, voiced a key concern: "I have to make sure that the agent doesn't take over the thinking of my junior engineer." This means junior engineers now need to "grow up" faster, asking critical design choice questions of agents—like why an agent chose a synchronous API over an asynchronous one. You can't just accept the output; you have to understand the reasoning. This has dramatically reshaped the hiring process. The days of whiteboarding bubble sort or simply testing coding speed are largely gone. Instead, companies are filtering for "high-caliber candidates" who can actually reason about AI-generated code. Kumo.ai, for example, now includes a take-home problem where candidates use an agent to code a solution, then come in to explain the agent's choices and justify them. The focus has shifted to reading code, understanding design decisions, and ensuring trust in the output through diligent testing. It's an elevation of the interview process, seeking engineers who are not just fast coders but critical thinkers and architects. The traditional definition of a senior engineer—someone who can take an open-ended problem, define it, and then solve it—remains, but it's now amplified. Whether designing a database for an infrastructure engineer or evaluating model assumptions for an AI specialist, the ability to ask the right questions about unspecified specs or product needs has always been the hallmark of senior talent. AI simply makes that skill more indispensable.

Future-Proofing Governance and Architecture

Looking ahead, the industry will inevitably build out a more robust governance layer. CIOs and CISOs will demand, and receive, tooling that provides clear visibility into API calls and data egress. As organizations onboard new vendors, "Can I actually go see where all my data is going?" will become a must-have question. Cost will also drive a tighter grip on governance. In this high experimentation phase, data and tokens are being sent "all over the place" without much thought. Expect engineering teams to start considering whether an open model, deployed internally within an organization’s perimeter, might be a more cost-effective and secure solution than relying on external services. These design choice questions will become central to engineering design reviews. It’s an incredibly exciting time to be in engineering, as Raghavan concludes. But while we embrace the dizzying pace of change and the opportunity to make "new mistakes," it's equally vital to internalize the lessons from past architectural blunders. We're building at blazing speed, yes, but avoiding the familiar pitfalls of unmanaged sprawl and technical debt will define the truly successful organizations in this AI-augmented future.