crossroads in the information world.
On one hand, there’s a universal recognition of the worth of internal data for AI. Everyone understands that data is the critical foundational layer that unlocks value for agents and LLMs. And for a lot of (all?) enterprises, this isn’t just another innovation project — it’s viewed as a matter of life or death.
Then again, “legacy” data use cases (business intelligence dashboards, ad-hoc exploration, and the whole lot in-between) are increasingly viewed as nice-to-have collections of high-cost, low-value artifacts. The C-suite and other data stakeholders are slowly but steadily beginning to ask the uncomfortable query out loud: (Well, fair enough.)
This puts data teams in a precarious spot. For the last five years, we invested heavily within the Modern Data Stack. We scaled our warehouses and treated every problem as a nail that needed a dbt hammer. (Because another dbt model will create all of the difference, right? Rigth?) We collectively convinced ourselves that surely more tooling and more code will end in more business value and happier data consumers.
The result? Unnecessary complexity and “model sprawl.” We built an ecosystem that was easier than Hadoop, sure, but we optimized for volume somewhat than value.
Today, data teams are paralyzed by mountains of tech debt — hundreds of dbt models, lots of of fragile Airflow DAGs, and a sprawling vendor list — while the business asks why we will’t just “plug the LLM into the information” tomorrow.
We were caught off guard. The killer use case finally arrived, and it’s more exciting than we ever anticipated, but our tooling was built for a special era (and critically, a special sort of data consumer). For a gaggle of people that work with predictions each day, we turned out to be terrible at predicting our own future.
However it’s not too late to pivot. If data teams wish to survive this shift, we’d like to stop constructing prefer it’s the height of the dbt gold rush. In this text, I’ll cover six strategic imperatives to deal with without delay, as you, fellow data person, transition to a very recent .
1. Features as Products, No More: Putting the Stack on a Eating regimen
This sounds counterintuitive, but hear me out: Step one to survival isn’t adding; it’s subtracting.
We want to have an honest (and barely uncomfortable) conversation about “Modern Data Stack” bloat. For a couple of years, we operated under a model where each feature an information team needed was a separate vendor contract. We principally traded configuration friction for bank card swipes. While the architecture diagrams we (myself included) designed during this era, featuring dozens of logos and a dedicated tool for each minor step within the pipeline, may need looked impressive on a slide, they created an ecosystem that’s hostile to quick iteration.
The landscape has shifted. Cloud data platforms (the Snowflakes and Databricks of the world) have aggressively moved to consolidate these capabilities. Features that used to require a specialized SaaS tool, from notebooks and light-weight analytics to lineage and metadata management, at the moment are native platform capabilities.
The need for a fragmented “best-of-breed” stack is becoming an anomaly, applicable only to area of interest use cases. For the masses, built-in capabilities are finally adequate (really!). In 2026, probably the most successful data teams won’t be those with probably the most complex architectures; they’ll be those who realized their cloud data platform has quietly eaten 70% of their specialized tooling.
There’s also a hidden cost to this fragmentation that kills AI projects: Context Silos.
Specialized vendors are notoriously (to say the least) of the metadata they capture. They construct walled gardens where your lineage and usage data are trapped behind limited (and barely documented) APIs. This, unsurprisingly, is fatal for AI. Agents rely entirely on context to operate — they should “see” the entire picture to reason accurately. In case your transformation logic is in Tool A, your quality checks in Tool B, and your catalog in Tool C, with no metadata standards in between, you could have fragmented the map. To an AI agent, a posh stack just looks like a series of black boxes it cannot learn from.
The Eating regimen Plan:
- Declarative Pipelines over Heavy Orchestration: Do you really want a posh Airflow setup to administer dependencies when capabilities like Snowflake’s Dynamic Tables or Databricks’ Delta Live Tables can handle the DAG, retries, and latency routinely? The “default” orchestrator layer is shrinking: It’s still relevant (and needed) in some cross-system steps, but 90% of the orchestration could be managed natively.
- Platform over Plugins: Do you would like a separate vendor simply to run basic anomaly detection when your platform now offers native Data Metric Functions or pipeline expectations? The closer the check is to the information, the higher.
- The Artifact Audit: We’ve spent years rewarding “shipping code.” This incentive structure led to a codebase of hundreds of models where 40% aren’t used, 30% are duplicates, and 10% are only plain flawed. It’s time to delete code. (You won’t miss it, I promise! Code is a liability, not an asset.)
- Built-in over Bolt-on: The “best-of-breed” overhead — the mixing cost, the procurement friction, and the metadata silos — is now higher than the marginal advantage of those specialized features. In case your platform offers it natively, use it.
Survival depends upon agility. You can’t pivot to support AI agents when you are spending 80% of your week just keeping the “Modern Data Stack” Frankenstein monster alive.
2. True Decoupling: Storage (and Data!) is Yours, Compute is Rented
For the last decade, we’ve been sold a convenient half-truth in regards to the “separation of storage and compute.”
Vendors told us: And while that was true for the (and the bill), it wasn’t true for the . Your data, while technically sitting on cloud object storage, was locked inside proprietary formats that only that specific vendor’s engine could read. Should you wanted to make use of a special engine, you had to maneuver the information: We separated the bill, but we kept the lock-in.
A Latest Ice(berg) Age:
For the brand new wave of information use cases, we’d like true separation. This implies leveraging Open Table Formats (long live Apache Iceberg!) to make sure your data lives in a neutral, open state that compute engine can access.
This isn’t nearly avoiding vendor lock-in (though that’s a pleasant bonus). It’s about AI readiness and agility.
- The Old Way: You would like to try a brand new AI framework? Great, construct a pipeline to extract data out of your warehouse, convert it, and move it to a generic lake.
- The Latest Way: Your data sits in Iceberg tables. You point Snowflake at it for BI. You point Spark at it for heavy processing. You point a brand new, cutting-edge AI agent framework at it directly for inference.
No migration. No movement. No toil.
To be clear, this doesn’t mean abandoning native storage entirely. Keeping your high-concurrency serving layer (your “Gold” marts) in a warehouse format for performance is wonderful. The critical shift is that your gravity (the source of truth, the history, etc. ) now resides in an open format, not proprietary ones.
This architecture ensures you’re future-proof. When the “Next Big Thing” in AI compute arrives six months from now (or less?), you don’t must rebuild your stack. You simply plug the brand new engine into your existing storage, with no “translator” or friction in between.
3. Stop Being a Service, Start Being a Product
The dream of “universal self-serve” was a noble one. We wanted to construct a platform where anyone could answer any data query and create elegant artifacts/visualizations, with 0 Slack messages involved. In point of fact, we regularly built a “self-serve” buffet where the food was unlabeled and half the dishes were empty.
Data teams are almost at all times understaffed. Attempting to win every battle means you lose the war. To survive, you will need to pick your verticals.
The Shift to Data Products:
As a substitute of shipping “tables” or “dashboards,” it is advisable ship Data Products. A product isn’t just data; it’s a package that features (but isn’t limited to):
- Clear Ownership: Who’s the “Product Manager” for the Revenue Data?
- SLAs/SLOs: If this data is late, who gets paged? How fresh does it actually must be?
- Success Metrics: Is that this data/product actually moving the needle, or is it just “nice to have”?
I’ve written extensively in regards to the mechanics of information products before — from writing design docs for them to structuring the underlying data models — so I won’t rehash the main points here. The critical takeaway for the following era is the mindset shift: This isn’t just in regards to the data team changing how we construct; it’s about all the organization changing how they eat.
So, where to begin? First, stop attempting to democratize the whole lot without delay. Discover the three business verticals where data can actually create a “quick win” — possibly it’s churn prediction for the CS team or real-time inventory for Ops — and construct a cohesive, high-quality product there. You construct trust by solving specific business problems, somewhat than spreading yourself thin across all the company.
4. Foundations for Agents: The Context Library
We’ve spent a decade optimizing for human eyes (dashboards). Now, we’d like to optimize for machine “brains” (AI Agents).
As data teams, we were collectively taken off guard by the emergence of enterprise AI: While we were busy buying yet more SaaS tools to create more dbt models for more dashboards (), the bottom shifted. Now, there’s a supercharged AI that’s hungry for “context.” The initial response within the space was a rush to portray this context as simply connecting an LLM to your warehouse and catalog and calling it a day.
On the surface, that approach may sound “”, sure. It can end in some nice demos and impressive 10-minute showcases at data conferences. However the bad (good?) news is that production-grade context is way, way more than that.
An AI agent doesn’t care about your neat star schema if it doesn’t have the semantic meaning behind it. Giving an LLM access to only breadcrumbs (whether it’s table/field names or a Parquet file with columns like attr_v1_final) is like giving a toddler a dictionary in a language they don’t speak. It drastically limits the sphere of possibilities and forces the LLM to hallucinate generic, low-value context to fill the huge void left by our collective lack of standardized documentation.
Constructing the Context Library:
The “Semantic Layer” has been an on-and-off hot topic for years, but within the AI era, it’s a literal requirement. Agents deserve (and require) way more than the skinny layer of metadata we’ve inbuilt the Modern Data Stack world. To get things back heading in the right direction, it is advisable start doing the “unglamorous” groundwork:
- The Documentation Debt: It’s not enough to know to calculate a metric. AI must know the metric represents, it’s calculated that way, and owns it. What are the sting cases? When should a condition be ignored? And most significantly, what must occur once a metric moves? (More on this later.)
- Capturing the “Oral Tradition”: Most business context currently lives in “tribal knowledge” or forgotten Slack threads. We want to maneuver this into machine-readable formats (Markdown, metadata tags, etc.) that detail how the business actually operates — from the macro technique to the micro nuances.
- Standards & Changelogs: Agents are highly sensitive to alter. Should you change a schema without updating the “Context Library,” the agent (understandably) hallucinates. Documenting means ensuring that your context is a living organism that accurately reflects the present state of the world and the events that led to it (with their very own context).
The format matters lower than the content. AI is great at translating JSON to YAML to Markdown (so definitely use it to bootstrap your context library from raw code and Google docs, supplying you with a solid baseline to refine somewhat than a blank page). It’s great, nevertheless, at guessing the business logic you forgot to write down down.
In brief: Document, document, document. The AI gods will determine how one can read your documentation later.
5. From “What Happened?” to “What Now?”
The pre-AI world was a passive, descriptive one. We called it BI.
The workflow went like this: You construct a dashboard, it sits in a corner, and a human has to recollect to take a look at it, interpret the squiggle on the chart, after which resolve to take an motion (or, way more continuously, just do what they were planning on doing anyway). That is the “Data-to-Decision” gap, and it’s where value goes to die.
In tomorrow’s brave recent world, the micro-decision will now not be taken by humans. Humans set the , sure, however the is getting automated at a powerful pace.
We want to stop being the team that “provides the numbers” and begin being the team that builds the systems that turn those numbers into immediate motion.
Architecting the Feedback Loop:
We want to shift from passive dashboards to automated feedback loops.
- Metric Trees over Flat Metrics: Don’t just track “Revenue.” Track the granular metrics that feed into it and map how they’re interconnected. The formula isn’t at all times exact or scientific, but capturing the relationships is critical. An AI agent must know that influences (+ how and why) to traverse the tree and find the foundation cause.
- The “If This, Then That” Strategy: If a granular metric moves outside of an outlined threshold, what’s the automated response? We want to encode this logic and different paths that align with the general business strategy. ( Churn risk for Tier 1 users spikes. A dashboard turns red. Someone possibly sees it next week. Trigger an automatic outreach sequence (with fine-tuned AI-powered messaging) and alert the account manager in Salesforce immediately.)
- Lively Navigation over Passive Validation: The industry remains to be unfortunately suffering from “Validation Theater”: using charts to retroactively justify decisions already made. Changing this dynamic is mandatory as AI becomes more capable. The goal is to construct systems where data acts as a strategic navigator: actively analyzing real-time context to propose the optimal path forward and, where appropriate, routinely triggering the following step (inside defined guardrails). The dashboard shouldn’t be a report card; it ought to be a advice engine.
The query isn’t “What does the information say?” It’s: “Now that the information says X, what motion are we taking routinely?”
6. The Evolving Data Persona: “Who Writes the SQL” Doesn’t Matter
Just a few years ago, the “Analytics Engineer” was essentially a dbt model factory. Today, that role is slowly evaporating as humans move one abstraction layer up in practically all professions. In case your primary value prop is “I write SQL,” you’re competing with an LLM that may do it faster, cheaper, and increasingly higher.
The info roles of the following wave will probably be defined by rigor, architecture, system pondering, and business sense, not syntax or coding skills.
The Full-Stack Data Mindset:
- Moving Upstream (Governance): We will now not just clean up the mess once the information reaches our clean and tidy data platform (is it?). We want to maneuver left by establishing Data Contracts (no matter format) on the source and enforcing quality at the purpose of creation. It isn’t any longer enough to “ask” software engineers for higher data; data teams need the engineering fluency to actively collaborate with product teams and construct data-literate systems from day one.
- Moving Downstream (Activation): We want to catch up with to the activation layer. It’s not enough to “enable” the business; we’d like to act as Data PMs, ensuring the information product actually solves a user problem and drives a workflow. (Thus, as an information person, understanding the business you’re constructing products for is quickly becoming a requirement.)
- Working Above the Code: Your job is to define the standards, the rules, and the governance. Let the machines handle the boilerplate when you make sure the business logic is sound and the AI has the appropriate context.
It doesn’t matter who (or what) writes the code. What matters is the rigor: Data mistakes within the AI era are exponentially more costly. A flawed number in a dashboard is an annoyance that, let’s be honest, gets ignored half the time. A flawed number in an AI agent’s loop triggers the flawed motion, sends the flawed email, or turns off the flawed server — routinely and at scale.
A final reality check: It’s all in regards to the business
After I transitioned from data engineering to product management a few years ago, my perspective on the information team’s role shifted immediately.
As a PM, I spotted I don’t care about neat data models. I don’t care if the pipeline is “elegant” or if the information team is using the best recent tool. I even have a gathering in quarter-hour where I would like to make a decision whether to kill a feature. I just need the information to reply my query so I can move forward.
Data teams are, by design, a bottleneck. Everyone wants a chunk of your time. Should you cling to “the best way we’ve at all times done it” — insisting on perfect cycles and rigid structures while the business is moving at AI speed — you will probably be bypassed.
The Survival Kit is ultimately about flexibility. It’s about being willing to let go of the tools you spent years learning. It’s about realizing that “Data Engineer” is only a title, but “Value Generator” is the profession.
Embrace the mess, cut the fat, and begin constructing for the agents. Over the following decade, the information landscape goes to be wild — be sure you’re not distracted by the impressive architecture diagrams or cool tech you see along the best way; the one consequence that matters will at all times be how much value you generate for the business.
Mahdi Karabiben is an information and product leader with a decade of experience constructing petabyte-scale data platforms. A former Staff Data Engineer at Zendesk and Head of Product at Sifflet, he’s currently a Senior Product Manager at Neo4j. Mahdi is a frequent conference speaker who actively writes about data architecture and AI readiness on Medium and his newsletter, Data Espresso.
