Quick recap from my previous article: In high-performing product organizations, strategy and execution aren't split between different roles. Every product team needs someone who owns both the "why" and accountability for what is built: someone with strong commercial instinct, ownership of outcomes, and direct connection to execution.

"But does this scale?"

Now let me address the inevitable objection: "Sure, but that integrated model might work at startup scale. Once you have hundreds of teams and complex enterprise environments, you need to split these responsibilities for scalability."

And while the scale at which you operate can influence your organizational design, a good principle is good at any size, and a bad principle is bad at any size.

There are several empirical examples of companies that scaled to massive size without splitting strategy from execution. Google doesn't have Product Owners. Neither does Cloudflare, nor Netflix. They scale not by fragmenting the role, but by managing scope: ensuring each Product Manager has a clear, bounded mandate where they can make meaningful impact without drowning in dependencies.

A sustainable model to scale Product teams

If you shouldn't split strategy from execution, how do you scale product management?

The answer is vertical, not horizontal. You scale by expanding scope, not by fragmenting responsibilities:

Product Manager → Owns a single product or module with clear boundaries. Responsible for both strategy and execution. Measured on business outcomes and impact. Works with a squad sized and staffed appropriately to the scope. Following Team Topologies principles, the squad must be optimized for fast flow and own the end-to-end value generation of a slice of the problem space. In startups, that slice will be very big. In huge corporates, that slice will be very small.

Group Product Manager / Head of Product → Owns a portfolio of products or a complex product with multiple independent modules. Sets portfolio-level goals. Acts as people manager for Product Managers. Provides coordination across products and platforms.

Director of Product / VP / CPO → Owns product organization vision, culture, tooling, and structure. Ensures product management practices scale effectively. Builds systems that empower PMs to make impact. In smaller companies who don’t need 2+ management layers, this role is usually merged with the previous one. In startups, this is usually the CTO (heading both the Engineering and Product functions).

In this structure, the Group PM isn't "doing strategy" while PMs "execute." Instead, the Group PM works at a higher level of abstraction—setting portfolio goals and ensuring alignment—while individual Product Managers maintain full ownership of both strategy and execution within their scope.

The Group PM asks: "Are we working on the right set of products to achieve our business objectives?"

The PM asks: "Is my product's roadmap delivering maximum impact on those objectives?"

Both are strategic. Both are connected to execution. The difference is scope, not separation of concerns.

Assigning the right scope

The key to scaling is assigning the right scope to each Product Manager. Not so broad that they're drowning in dependencies and can't make meaningful decisions. Not so narrow that they lack autonomy or can't see how their work connects to business outcomes.

When you get the scope right and each PM has a clear mandate and a squad-sized team to execute it, you can scale this model from dozens to hundreds of product teams without introducing artificial role splits.

What does "right scope" look like?

• Clear boundaries: The PM can make most decisions without constant cross-team coordination. There are no debates about ownership and no handoffs between teams in the value chain.

• Measurable impact: Success can be tied to specific business outcomes, not just feature completion.

• The right team: The PM works with a squad that can execute the roadmap without being stretched too thin or sitting idle. The squad’s combined skills allow the team to ship meaningful work without having to delegate work to another team.

• Strategic autonomy: The PM has enough freedom to experiment and adjust course based on learnings. The squad’s ability to ship new features is not constrained by heavy dependencies on other teams.

The takeaway: scaling is about scope, not splitting

Scaling is possible even with a single person managing both product strategy and execution. The most important thing is setting the scope right and using the management chain to coordinate overarching topics.

If you’ve been following the broader “Product Owner vs Product Manager” debate from the previous article, it can be tempting to treat scaling as a question of titles and role boundaries.

But the pattern underneath is the same: organizations split “strategy” from “execution” when they’re trying to compensate for a lack of true outcome ownership.

So the leadership takeaway is this: companies get the product management they incentivize. If you measure people on throughput, compliance to process, and “staying on plan,” you’ll get backlog managers and ceremony owners, regardless of whether you call them PMs or POs. If you give clear scope and accountability for impact, you’ll get product leaders who can hold the whole loop: problem → delivery → learning → outcomes.

When you get that right—scope and incentives aligned—your product organization can scale to hundreds of teams without ever needing to split product strategy from execution.

Legacy migration has an almost universal playbook:

• Step 1 - Asset Discovery: Make an export of all assets in the environment to see what is there to migrate. The output is usually a massive spreadsheet or dashboard. The leadership sees the number and it becomes the anchor: We have X thousand resources to migrate.

• Step 2 - Categorize: Tag the assets based on their domain, owner and the type of migration they potentially require. This is usually done manually.

• Step 3 - Assign: Now that we know what we have, and who owns it, it's time to assign each slice to a team and say: These are yours. Assess them, decide what to do with each one, and report back.

• Step 4 - Teams Investigate and Act: Each team is expected to review their assigned assets, determine what's still needed, plan the migration, rewrite what's valuable, and flag the rest for deletion.

• Step 5 - Track and Report Progress: A program manager tracks completion rates. Dashboards show how many assets have been categorized, how many migrated, how many deleted, and the progress is measured as a percentage of the original inventory.

The Hidden Assumption

Every step in this playbook sounds reasonable on its own. Together, they rest on assumptions that rarely hold:

Every asset has an owner

The plan assumes that for each resource, someone in the organization knows what it is, why it exists, and whether it still matters. In practice, organic environments accumulate resources that outlive the teams or individuals who created them. The owner left, the project ended, but the assets are running and getting billed.

Assets map to products or business domains

The plan assumes you can draw lines from resources to business capabilities:

• these tables belong to marketing analytics.

• those belong to finance reporting.

• ...

In reality, only a small percentage of assets map cleanly to such groups. Most are organized around pipelines, ad-hoc SQL, and intermediate computation steps, not around products.

Inventory equals understanding

This is the deepest assumption. Listing everything feels like progress. But any environment painful enough (in cost or maintenance) to justify migration has accumulated significant scale. The inventory will contain hundreds of thousands of assets with no inherent grouping. It feels like the first logical step. It's actually the first illusion of control.

Teams can self-serve classification

The plan assumes you can hand teams a filtered list and they'll sort it out: keep, migrate, delete. But it requires:

1. Teams actually recognize the assets. Which usually means reverse-engineering their history.

2. The rest of the organization, dependent on those resources, holds still. But what if they change their process during the migration, and with it, their requirements?

Effort scales with volume

The intuition is: twice as many assets, twice as much work. But in reality effort scales with ambiguity. A thousand well-structured, well known assets might be easier to migrate than a handful of orphaned, unnamed ones. The cost is in the investigation and understanding, not the count.

What You Actually Find

I recently faced this exact situation, and the reality was eye-opening. A data warehouse that had accumulated tables, data pipelines, and scheduled queries over the past 10 years. The tables alone numbered over half a million.

No reasonable grouping would make that number workable. Not even if every technical member focused solely on migration and parked all other work.

So I asked a different question: do we really need to migrate all of this? Asking teams wasn't an option, for all the reasons above. But I could observe the environment directly. Are any of these tables actually being read?

And the result was surprising. Over 99% of the tables hadn't been accessed in the past 90 days. A large portion had never been accessed at all. Of those that were accessed, only a small percentage showed consistent, ongoing usage.

We stopped asking teams to review the full inventory. The migration was not about moving half a million assets to a new platform. It was about finding the ones that were still alive.

Why Understanding IS the Migration

The default playbook frames migration as logistics: X thousand assets need to move from A to B, and progress is the percentage that have moved. This treats every asset as a unit of work. But the real unit of work is not the asset. It is the decision: does this asset still matter?

That reframing changes the shape of the project. A logistics project scales with volume or basically twice the assets, twice the effort. A decision project scales with ambiguity. And ambiguity is not evenly distributed. In our case, one query against 90 days of usage data made the decision for 99% of the environment. No team meetings, no reverse-engineering, no spreadsheets. The remaining 1% was the only part that needed human judgment at all.

This is why inventory-first feels right but leads to pointless activities. An inventory gives leadership a number they can put on a slide: 500,000 assets to migrate. But that number tells you nothing about how many decisions you actually face. It treats an orphaned table untouched for five years the same as a pipeline feeding a daily business report. Usage data makes them obviously different. The inventory makes them equal.

The default playbook also misidentifies what a legacy environment is. It assumes a system something designed, something you can pick up and relocate. But a ten-year-old data warehouse is not a system. It is an accumulation: layers of decisions made by people who are no longer around, solving problems that may no longer exist. You do not relocate an accumulation. You find what is still alive inside it and build forward from there. Martin Fowler describes the general pattern as a Strangler Fig the new system grows around the old one while the old one atrophies. But in a mature legacy environment, most of the atrophy has already happened. The tables are already dead. The pipelines already stopped. You just haven't confirmed it yet.

Understanding the environment does not prepare you for the migration. It is the migration. Once you know what is alive, the rest is cleanup.

An Alternative: Quarantine Before You Classify

The alternative requires a different starting point: let usage tell you what matters, instead of asking teams to figure it out from a spreadsheet.

Start with what you can act on immediately. Usage data (who accessed what, and when) separates the living parts of the environment from the dead ones, without requiring anyone to understand what each asset is.

Once you have that separation, introduce a quarantine window. If a resource hasn't been accessed for a defined period (say 90 days), block access to it and notify the teams. If nobody requests restoration within another 90 days, back it up and delete it. If someone does need it, you've just identified a genuinely valuable asset: route it to the appropriate team and start a proper migration lifecycle for it.

The important assets identify themselves. Instead of teams sifting through hundreds of thousands of items, the environment surfaces its own priorities through actual usage. The noise falls away on its own.

Postpone permanent deletion as long as practically possible. Data loss is the one mistake you can't reverse. Quarantine simulates deletion before you commit to it, and catches seasonal patterns that a 90-day snapshot would miss.

With this model, you can report meaningful progress monthly: deadwood percentage, reduction in active resources, cleanup projections. Metrics an executive can act on, without requiring teams to manually work through irrelevant assets.

The Question to Ask About Your Own Environment

Before committing to the default playbook, ask one question about your legacy environment: what percentage of your assets have been accessed in the last 90 days?

If the answer is low, and it may be far lower than you expect, you are not facing an inventory problem. You are facing an archaeology problem. Archaeology does not start with a catalog. It starts with a question: what here is still alive?

This document explains how we use Claude at DataChef.

1. Always commit your claude.md

Every project that uses Claude must have a claude.md.

This file explains how Claude should behave in the project.

It should include things like:

• coding style

• architecture rules

• libraries we prefer

• things we never do

• how tests should look

• how commits should look

Why this matters:

Claude works best when it has context. Without context it guesses. With context it becomes consistent.

Your claude.md is the memory of the project.

Commit it to the repository so everyone works with the same rules.

2. Put your important prompts in the pull request

When Claude helps write code, the reviewer should know how that code was created.

If a prompt had a big influence on the result, include it in the pull request description.

This helps reviewers understand:

• the intent of Claude

• the reasoning of Claude

• what Claude was asked to do

It also makes it easier to reproduce or improve the result later.

Transparency builds trust.

3. Teach Claude not to sound like AI

Add a new skill for you CLaude to not sound like AI and avoid these patterns:

https://en.wikipedia.org/wiki/Wikipedia:Signs\_of\_AI\_writing

AI text often looks like:

• overly formal language

• repetitive structures

• too many bullet points

• vague explanations

• generic transitions

We want writing that sounds human and direct.

Short sentences. Clear thinking. No filler.

4. Never use dangerously skip permissions for Production

Do not start Claude with --dangerously-skip-permissions

Always review what Claude wants to do.

Read each prompt and tool request before approving it.

You should always know:

• what files Claude reads

• what files Claude changes

• what commands it runs

5. Claude writes code. You own the code.

Claude can generate code, but you are responsible for it.

Always:

• read the code

• understand the code

• question the code

If you cannot explain a change to another developer, do not merge it.

6. Ask for small steps

Do not ask Claude to build a whole system in one prompt.

Work in small steps.

Example flow:

1. ask Claude to design the approach

2. review the plan

3. implement one part

4. review again

5. continue

Small steps reduce mistakes.

7. Prefer editing over generating

If a file already exists, ask Claude to improve or refactor it.

Do not ask it to rewrite everything.

Large rewrites often introduce hidden problems.

Good prompts look like:

• "simplify this function"

• "remove duplication"

• "add tests for this logic"

• "explain the edge cases"

8. Always ask for tests

If Claude writes logic, it should also suggest tests.

Tests help verify that the code does what we expect.

Good prompts include:

• "write unit tests for this function"

• "add edge cases"

• "show failure scenarios"

You can add this as part of your CLAUDE.md to make sure you never forget.

9. Ask Claude to explain its reasoning

Before accepting a change, ask Claude questions.

Examples:

• why is this approach better

• what edge cases exist

• what could break

• what are the performance risks

Claude is good at surfacing hidden issues when asked directly.

10. Keep prompts simple and direct

Claude works best with clear instructions.

Bad prompt:

"Can you improve this in a robust scalable architecture that follows best practices?"

Better prompt:

"Reduce complexity in this function. Do not change the behavior."

Clarity produces better results.

11. Use Claude as a thinking partner

Claude is not just for writing code.

It is useful for:

• debugging

• reading unfamiliar code

• designing APIs

• writing migrations

• reviewing pull requests

• explaining errors

Treat it like a second developer who thinks fast.

But remember: you are the final reviewer.

12. Leave the codebase better

Every Claude assisted change should improve the codebase.

Examples:

• clearer naming

• better structure

• fewer lines

• stronger tests

• simpler logic

Speed is helpful. Quality is the goal.

Final rule

Claude is powerful.

But good engineering still comes from:

• careful thinking

• good reviews

• clear communication

• responsibility for the code

Use Claude to move faster.

Do not use it to stop thinking.

No more bottlenecks. No more waiting. Just ask DANA (Data and AI knowledge Assistant)

The Problem: One Expert, Ten Teams, a Hundred Questions

Large organizations undergoing system migrations or platform integrations know this scenario well. A domain expert owns the knowledge about how the data model works. Business analysts across multiple downstream teams need answers to move forward:

• "I'm consuming a field from the old system. What's the equivalent in the new data model?"

• "Can this field be empty, or is it always required?"

• "What does this status code mean and how should I use it?"

• "What are the right fields to get the official name at different hierarchy levels?"

These are the kinds of questions that come up dozens of times a day during a migration. The answers exist, but they're scattered across Confluence pages, Excel mappings, design documents, and most critically, in the domain expert’s head.

The architect spends their days answering repetitive questions instead of doing architectural work. Analysts wait hours or days for responses. Migration timelines slip. And when the architect is unavailable, progress stops entirely.

💡This is not a tooling problem. It's a knowledge bottleneck. And you can't solve it by buying another documentation tool that nobody will maintain.

Why Existing Tools Don't Solve This

You might think: "We already have a data catalog" or "We documented everything in Confluence." The reality is that most organizations have tried at least one of these approaches, and they all fall short in the same way.

Data catalogs are great for storing metadata, and many support field descriptions, glossary links, and even column-level lineage. But that information still needs to be written and maintained by someone. And when it comes to cross-system mappings during a migration, how a field in the old model translates to the new one, under what conditions, and with what edge cases, catalogs typically don't capture that level of contextual knowledge.

Documentation wikis (Confluence, SharePoint, Notion) become outdated the moment they're written. Maintaining them requires manual effort that nobody has time for. After a few months, teams stop trusting them and go back to asking the domain expert directly.

Schema extraction tools can pull table structures and column types from a database. But CUST_FLG_02 is still just CUST_FLG_02. Without a business context, raw schema data is only marginally useful.

Documentation sprints work in theory: you get everyone in a room, write everything down, and publish it. In practice, the knowledge is stale before the sprint is over, and you've burned weeks of your domain expert's time in the process.

The common thread? All of these approaches treat knowledge capture as a one-time event. But domain knowledge evolves constantly, especially during migrations. You need something that learns continuously and stays up to date.

What is DANA?

DANA is a conversational AI that sits between your domain experts and your teams. Instead of trying to replace the architect, DANA learns from them through natural conversation, and then makes that knowledge available to everyone on demand.

Think of it as a colleague who has perfect memory. The domain expert explains something once, and DANA remembers it, structures it, and can repeat it accurately to anyone who asks.

But unlike a wiki page, DANA doesn't just sit there waiting to go stale. It actively keeps itself up to date. When team members ask questions that reveal gaps, DANA flags those gaps and prompts the domain expert to fill them. When new schemas are detected or data models change, DANA identifies what's outdated and initiates a review. The architect doesn't need to remember to update DANA, DANA tells the domain experts what needs attention.

DANA combines three things that are usually separate: schema extraction (understanding the technical structure of your data), knowledge enrichment (adding business context and meaning), and a conversational AI interface (so humans can teach it and query it naturally). Together, these create a knowledge base that isn't just a static document, but a living system that grows smarter with every interaction.

How DANA Works

1. The Architect Teaches, DANA Remembers

A data expert talks to DANA the same way they'd talk to a colleague. They explain how the data model works, clarify field meanings, describe mappings between old and new systems, and flag edge cases. DANA captures these explanations, structures them, and stores them as validated knowledge.

No documentation writing. No spreadsheet filling. Just conversation.

Here's what a session might look like:

Architect: "The product code field in the legacy system maps to a different field name in the new domain model. It's always populated for active records, but can be null for discontinued items."

DANA: "Got it. I've recorded the mapping between those two fields. Populated for active records, nullable for discontinued. Is there a specific status field I should check to determine if a record is discontinued?"

Architect: "Yes, there's a lifecycle status field. If it's set to 'DISCONTINUED', the product code may be null."

DANA: "Understood. I've linked this to the lifecycle status field as a dependency. Changed: product code mapping documented with nullability rule. Gaps: I still don't have mappings for 3 other attributes in the pricing group. Next: Want to work on those, or switch to a different area?"

DANA doesn't just record what you say. It asks follow-up questions, connects related facts, tracks what's still missing, and tells you what to work on next.

Rather than uploading large volumes of raw internal data, the architect teaches DANA the concepts and mappings it needs. This keeps sensitive client data secure while still building a rich, contextual knowledge base.

2. Schema Understanding

DANA also ingests schemas directly from your databases and warehouses. It connects to systems like Oracle, SQL Server, Teradata, and PostgreSQL, and extracts the technical picture: tables, columns, data types, primary and foreign keys, stored procedures, and views.

But DANA goes a step further than typical schema extraction. It also analyzes query execution history to understand which parts of the data are actually in use. Which tables are queried daily? Which ones haven't been touched in years? Which columns show up in the most joins?

This usage profiling helps teams prioritize. Instead of trying to document every table in a legacy system with thousands of objects, you start with the ones that actually matter to the business.

3. The Team Asks, DANA Answers

Once DANA has learned from the architect, anyone on the team can ask questions directly through the conversational AI interface. And they get real answers, not just links to documents.

Here's what that looks like in practice:

Data Engineer: "I'm consuming the product code from the legacy system. What is the equivalent attribute in the new domain model?"

DANA: "The equivalent field in the new model is the product identifier. Note: this field can be null for discontinued records (based on the lifecycle status). For active records, it's always populated.

Source:

• domain architect name

• relevant document links"

DANA provides answers with references to where the knowledge came from.

Domain experts teach DANA what they know. Downstream teams consume that knowledge on demand. DANA tracks what it has learned, what is still missing, and what needs validation, creating a living knowledge base that improves with every interaction.

4. DANA Evolves With Your Data

DANA's knowledge isn't a snapshot, it evolves as your data model evolves. Every question from the team that can't be fully answered becomes a signal: a gap that DANA tracks and raises with the architect in the next session. When schemas change, or new source systems are added, DANA detects what's new and flags what needs to be reviewed or remapped. This creates a continuous feedback loop: the team's questions drive the architect's priorities, and the architect's answers expand what the team can self-serve. Over time, DANA covers more ground, answers more questions, and requires less input from the architect.

What Your Teams Get

• Instant answers: No more waiting for the architect's calendar to open up. Teams get reliable, referenced answers through the conversational AI interface whenever they need them.

• Consistent knowledge: Everyone gets the same validated information. When the architect clarifies a mapping with DANA, that clarification is immediately available to all teams.

• Reduced key-person risk: Domain knowledge is no longer locked in one person's head. It's captured, structured, and accessible, even when key people move on.

• Faster migrations: Impact analysis that used to take weeks of back-and-forth can happen in days when teams can self-serve.

• Data contracts: DANA generates YAML-based data contracts following the datacontracts.com specification. A data contract defines the structure, meaning, ownership, and rules of a dataset in a machine-readable format. Think of it as an API contract, but for data. These contracts can be versioned in Git, published to your data catalog, and used to enforce data quality standards across teams.

Who is DANA For?

DANA is built for organizations where data knowledge is the bottleneck:

Conclusion

Your domain experts shouldn't be answering the same questions over and over. And your teams shouldn't be waiting in line for answers that already exist somewhere.

DANA turns conversations into knowledge, and knowledge into self-service. It learns continuously, stays consistent, and scales to every team that needs it.

That's conversational AI applied to data. And it changes how organizations approach legacy modernization.

Interested in DANA or facing a similar challenge? Reach out to us at datachef.co/contact or connect with us on LinkedIn.

#conversational-ai #data-engineering #data-contracts #legacy-modernization

dbt exposures are a lightweight way to document those consumers as code, right next to the models and sources they depend on. They turn “tribal knowledge” into something reviewable, queryable, and much harder to accidentally forget.

Why does “docs outside the repo” drift

Those who have worked with me might know that I take a very careful approach when it comes to documentation, especially when it is decoupled from the codebase. Reasons include:

• When someone is reading a piece of code, they might not think of looking elsewhere (Confluence, SharePoint) for extra insight.

• You can be certain that shortly after the documentation is published, it will become outdated because code contributors forgot or did not prioritize documentation updates.

On the other hand, providing context about logic someone went through the trouble to implement provides enormous value. In the case of dbt models: why do we have a model? Who or what consumes it? What are the consequences if it breaks, or if we remove or alter it? Who should we reach out to before we clean up a legacy model?

An intermediate model (i.e. which is upstream of another model) has an obvious purpose. Its usage is self-documented: just look at its children models. But what about all the models dangling at the far right end of the DAG in your dbt project?

Most users are aware that models and data sources can be enriched with documentation and descriptions. But dbt also offers a powerful way to document the usage of models as code. This feature may not be well known, so in this post I will explain it and (hopefully) convince more dbt practitioners to adopt it.

Defining exposures in YAML

In your dbt project, you can add one or more YAML files that declare exposures. Let’s first look at how to do it (slightly modified example from the [official documentation](https://docs.getdbt.com/docs/build/exposures)):

exposures:

  - name: weekly_jaffle_metrics
    label: Jaffles by the Week
    type: dashboard
    maturity: high
    url: https://bi.tool/dashboards/1
    description: >
      Did someone say "exponential growth"?

    depends_on:
      - ref('fct_orders')
      - ref('dim_customers')

    owner:
      name: Callum McData
      email: data@jaffleshop.com

Once you add exposures and generate docs dbt docs generate), they appear as first-class entities in the dbt documentation website: you can browse them, see their descriptions, owners, and links (for example, to the dashboard), and navigate their lineage to the upstream models and sources they depend on. In other words, they show up alongside models and sources in the docs UI, instead of living in a separate wiki.

Keeping documentation from drifting

An advantage of dbt exposures over, for example, a Confluence page is that they live close to the code. If a developer comes across a model, they can find the exposure easily because it sits in the same repository as the models. Thanks to this proximity alone, it is less likely to go out of sync. But another characteristic of exposures makes them even less likely to drift away from reality:

If a model referenced by an exposure (in this case fct_orders or dim_customers) is deleted or renamed, the dbt project won't be able to compile:

% dbt test

Encountered an error:

Compilation Error

Exposure 'exposure.weekly_jaffle_metrics' (models/exposures.yml) depends on a node named 'fct_orders' which was not found

This way, the contributor will be alerted that this model is being used and that care should be taken before going ahead with the change. This is because exposures are nodes in the graph of the dbt project, just like models and sources. If you reference a missing node, the project can’t compile.

So far, this probably sounds like exposures “solve” the problem. They help a lot, but they are not a silver bullet.

Limitations (and what exposures are not)

Exposures are useful, but it helps to be explicit about their limits:

• They do not catch all breaking changes. A column rename or semantic logic change can still break a dashboard even if the exposure compiles.

• They document known consumers. The absence of an exposure does not prove that a model is unused. It may simply mean the consumer has not been documented yet.

In other words, the developer’s vigilance is still required. In the next sections, I will show how exposures augment that vigilance by letting you query the graph for impact analysis (starting from an upstream model or source) and for investigations that start from a consumer (like a dashboard).

Impact analysis: find downstream consumers

If a source or model with many nodes downstream has an issue, or needs to be modified, we can query the dbt project’s graph to find all exposures downstream of that node, and therefore potentially affected. This gives you the impact of a change, plus a list of consumers and contact people to reach out to.

Let’s say the raw_payments source table (here a seed) has an issue, a raw_payments+ query shows the whole lineage down to the affected exposures:

If you prefer the CLI, this is a command that lists all exposures downstream of the raw_payments node:

% dbt ls --select raw_payments+ --resource-type exposure
exposure:jaffle_shop.customer_health_dashboard
exposure:jaffle_shop.customer_segmentation_ml
exposure:jaffle_shop.weekly_revenue_report

When the alert comes from a dashboard

Not every incident starts at the source. Sometimes a business user reports that a KPI in a dashboard looks off, or that a scheduled report has stopped refreshing. In that moment, the first question is usually: what are all the upstream models and sources that could explain this symptom?

If you declare dashboards (and other downstream assets) as exposures, they become a natural entry point for that investigation. Because each exposure lists the dbt nodes it depends on, you can quickly get an initial shortlist of models to inspect first.

For example, to find the nodes declared as dependencies of a given exposure (can also be used in dbt docs):

# List the dependencies of a specific exposure

% dbt ls --select +exposure:jaffle_shop.customer_health_dashboard

From there, you can keep traversing upstream, or combine it with other selectors to narrow your search. The key point is that you can start from “the thing that looks wrong” and move left through the graph, instead of guessing where to begin.

Review workflow: keep consumers involved

When introducing or modifying an exposure file, it's crucial to open the pull request like any other code change and request as a reviewer an owner of the consumer that's described in the exposure file. That way, that person can verify if we understood well how the model is used and that the contact information is correct.

This review step is also the best moment to make the description field truly useful. The consumer can help make it complete by adding the business meaning (what question this dashboard answers), important caveats (filtering rules, known limitations, edge cases), and freshness expectations (how often it should refresh, what “stale” means, and what to do when it is late). Over time, this turns exposures into a lightweight shared contract between producers and consumers.

This can be enforced with GitHub if exposures are defined together in a file for a group of users (a team): set that team as a codeowner on the file and their approval will be required if it’s modified.

Wrap-up

I hope this post has convinced you that exposures can benefit your dbt setup. They help document model usage, catch some mistakes early, and get a complete view of impacted consumers in case of data issues or code changes, all while staying up to date.

If you want a low-effort way to get started, pick one critical dashboard (the one people will notice within minutes if it breaks) and add a single exposure for it.

• Link to the dashboard.

• Add a real owner (a person or a team).

• Write a description that captures the business meaning, caveats, and what “fresh” means.

Once that is in place, you have a reliable starting point for impact analysis, and for investigations that begin from a consumer (“this dashboard looks wrong”).

On the contrary, people who've spent their entire career in large enterprises treat it as second nature. “You can't have a product team without a Product Owner!”, I heard several times. They might have more doubts about the Product Manager role, though, assuming they've ever worked with one.

So, is this just another article debating the differences between the two roles and whether you need one or both? Not quite. For me, the Product Manager vs Product Owner debate is asking the wrong question.

Yes, there's plenty of literature on the definition for each role and the right terminology to use. But here's the thing: in any organization that draws a hard line between these roles and wastes time debating where to draw it, the distinction itself is a symptom of organizational dysfunction, not a solution.

The real question isn't "what's the difference between these roles?" It's "why do so many enterprises struggle to build truly product-driven organizations?"

How we got here

Let's start with the facts. Product Owner is a role defined by Scrum, a specific methodology with a specific scope. In the Scrum framework, the Product Owner manages the backlog, prioritizes work, plans sprints, and communicates delivery status to stakeholders. Within its intended scope (execution within a squad) this is a perfectly valid role.

The Product Owner role proliferated alongside the adoption of scaled agile frameworks like SAFe, particularly in large enterprises trying to bring structure to dozens or hundreds of product teams. It became a way to ensure every squad had someone accountable for "what gets built next."

The role serves a purpose for companies that embrace Scrum, but it has significant limitations. With such a strong emphasis on operational excellence and delivery, Product Owners tend to focus on outputs rather than outcomes. They become skilled at managing sprints, refining user stories, and keeping stakeholders informed, but they rarely step back to ask whether the work delivers real value to customers or moves key business metrics. That strategic component is often neglected or taken for granted.

When things go south: the anti-patterns

Many enterprises tried to address the Product Owner role's limitations by introducing a top-down approach to "inject" that strategic component into teams. Since this is a gray area in Scrum methodologies, I've seen at least two major anti-patterns, each dysfunctional in its own way.

Anti-pattern #1: The Role Proliferation

Some enterprises create a two-tier system: Product Managers who set strategy, and Product Owners who execute it. On paper, this sounds like a clean division of labor. In practice, it's often a disaster.

The strategist loses touch with reality. When you're not involved in day-to-day prioritization and delivery, you lose the essential feedback loop that tells you whether your strategy actually works. You miss the technical constraints, the user journey friction, the implementation edge cases that should inform strategic decisions. For more complex products, you might even lose touch with the product itself. You are too distant from its users to develop empathy with them.

The executor loses agency. When you're handed a strategy from above and told to "just execute it," you become an order-taker. You can't make intelligent trade-offs because you don't fully understand the commercial reasoning behind the work. You're managing a backlog, not owning outcomes.

Anti-pattern #2: The Project Management Masquerade

Other companies go a different route: they only have Product Owners, and these Product Owners report to technical leadership, such as engineering directors, rather than product leaders.

This usually happens in IT teams within organizations undergoing a "transformation" program of some kind. Usually, in this kind of teams, there is no product discovery practice, and work originates from strategic initiatives handed down from above. In reality, these are glorified project management organizations where the Product Owner's job is simply to keep projects on track.

The Product Owner is never in charge of setting direction. They don't measure value, they measure compliance to a plan laid out by managers or, in the worst cases, by "the business."

And here's the telltale sign of this dysfunction: when someone in your IT organization refers to another part of the company as "the business," you have a problem. Because the Product Owner is supposed to be the business. They should understand the business's interests, contribute to them, and be evaluated on business outcomes, not (just) on timely delivery.

In this model, the Product Owner becomes a translator between stakeholders and engineers, a schedule tracker, a meeting organizer. They're valuable, but they're not product managers. And the organization doesn't have anyone truly owning the "why."

The model that actually works

Both anti-patterns stem from the same flawed assumption: splitting product strategy from execution and delegating strategy to a Product Manager, the management team, or another part of the organization.

Strategy and execution are not separate jobs. They are two sides of the same coin. The person making strategic bets needs to be close enough to implementation to course-correct. The person prioritizing daily work needs enough strategic context to make smart trade-offs without having to escalate every decision.

The split is artificial and creates more problems than it solves. It's far more natural for every product team to have one person who owns the "why" while staying accountable for what gets built. That role is usually called Product Manager, but the name doesn't matter—what matters is clarity about their skill set and responsibilities. Here's what they should bring to the table:

Strong commercial instinct. They're obsessed with customers and users, but also with business outcomes. They constantly ask: Is this work helping us capture new customers? Retain existing ones? Cut our operational costs?

Ownership of the "why." They own the problem space. What problem are we solving next? Why is this the most important thing? How will this impact the customer journey? And how will that translate to business outcomes? They can engineer a 'Yes' when a critical opportunity surfaces mid-sprint, finding creative ways to pivot without creating chaos.

Direct connection to execution. They work closely with engineers, understand technical constraints, and get their hands dirty by looking at data and analytics themselves. Not to micromanage, but because you can't make good strategic decisions without this ground truth.

All of the above is non-negotiable. Whether you're building data products, AI products, or consumer applications, this function must exist.

But it’s just as important to set boundaries and define what this person should not be burdened with:

• Writing detailed user stories (let tech leads translate PRDs into engineering work instead).

• Managing team logistics and sprint ceremonies.

• Tracking vanity metrics like velocity and story points.

• Managing people, hiring, writing performance reviews, etc.

These responsibilities dilute focus from what matters: understanding customer problems and driving business impact.

Outcomes, not backlogs

Stop debating whether to call people Product Managers or Product Owners. Start asking whether they own outcomes or just backlogs.

If someone on your team is responsible for a product but spends most of their time managing sprint logistics and tracking velocity, you have a process coordinator. Regardless of title.

If someone is setting product strategy but hasn't talked to a customer or used their own product in weeks, you have a strategist in an ivory tower. Regardless of title.

If someone takes orders from "the business" and measures success by on-time delivery and budget rather than impact, you have a project manager. Regardless of title.

What every product team needs is someone who owns the complete loop: from problem identification to solution delivery to impact measurement. Someone who thinks commercially, understands user journeys, and has the courage to say yes to the right opportunities even when it disrupts the plan.

Call them Product Manager. Call them Product Owner. Call them Chief Problem Solver.

Just don't split the job in half and expect either piece to succeed.

A year ago, if you told us we'd ship a production web application, 8 major features, 11 releases, 17,000 lines of TypeScript across a full-stack React + Express codebase with AI writing virtually all the code, we'd have smiled politely and moved on. But that's exactly what happened. And the interesting part isn't the AI. It's what we put around the AI.

The old way is gone (and that's okay)

Let's be honest: writing every line of code by hand is no longer the most productive way to build software. AI coding assistants have gotten remarkably good. They can scaffold components, implement API endpoints, refactor patterns across files, and even reason about architecture. We've reached a point where the bottleneck isn't typing code, it's knowing what to type.

And that's where most AI-assisted development goes sideways.

Hand an LLM a vague prompt like "build me a dashboard," and you'll get something. It might even look impressive for five minutes. But it won't match your design system. It won't follow your project's conventions. It won't handle the edge cases your users will inevitably find. And good luck maintaining it three months later when nobody (including the AI) remembers why certain decisions were made.

We needed a better approach. We needed to trust the AI, but with guardrails.

Enter spec-driven development

The idea is simple: don't start with code. Start with a specification.

Before a single line of code gets written, every feature goes through a structured pipeline:

1. Specify: Describe what you want in plain language. The system generates a formal specification with user stories, functional requirements, success criteria, and acceptance scenarios.

2. Clarify: The spec gets challenged. Are the requirements testable? Are there ambiguities? Edge cases? Up to 5 targeted questions are asked and the answers get encoded back into the spec.

3. Plan: Technical architecture gets designed. Research is conducted on the best patterns for each problem. Data models, API contracts, and integration scenarios are documented. Every decision gets a rationale and alternatives considered.

4. Tasks: The plan gets broken down into a precise, dependency-ordered task list. Each task has an ID, exact file paths, parallel execution markers, and belongs to a specific user story. Sequential vs. parallel execution is explicitly defined.

5. Implement: Tasks are executed phase by phase, respecting dependencies. Parallel tasks run simultaneously. Each completed task gets checked off. TypeScript compilation and tests are verified at the end.

This is what spec-kit does. It's an open-source specification framework that turns natural language into structured, validated, implementation-ready specifications, designed specifically for AI-assisted development. We adopted it at DataChef for one of our projects, and the results surprised us.

What this looks like in practice

Here's a real example from our project. We needed to replace pagination with infinite scrolling across 5 different table views. Instead of diving into code, we started with:

"Remove the paginations and load all data in every table, but keep the scroll inside the table, not for the whole page."

That single sentence went through the spec-kit pipeline and produced:

A specification with 2 user stories, 8 functional requirements, and 6 success criteria A research document with 5 technical decisions (CSS strategies for sticky headers, flex layout patterns, data volume analysis) API contracts documenting exactly how 4 endpoints would change A task list of 14 tasks across 3 phases, with explicit parallel execution groups A quickstart checklist with 30+ manual verification items

The AI then executed all 14 tasks (removing pagination from shared types, simplifying React Query hooks, updating 5 page components with sticky headers and scroll containers) and produced zero TypeScript errors and zero test failures.

The whole feature, from English sentence to working code, followed a traceable path where every decision was documented.

Why guardrails matter more than prompts

Here's what we learned after shipping 8 features this way:

Specifications are the real prompt engineering. A well-structured spec with clear acceptance criteria gives AI everything it needs to generate correct code. We stopped tweaking prompts and started improving specifications.

Research prevents costly mistakes. For one feature, the research phase discovered that our data volumes (under 200 records per table) didn't justify the complexity of incremental loading. That decision (documented with rationale) saved us from over-engineering. Without the research step, the AI would have happily built an unnecessarily complex infinite scroll system.

Parallel task execution is a superpower. Because spec-kit identifies which tasks touch different files, multiple AI agents can work simultaneously. In one phase, 5 agents updated 5 different page components in parallel. What would have been sequential 20-minute work completed in under 3 minutes.

The spec is the documentation. Every feature has a specs/ directory with its specification, plan, research decisions, API contracts, and task history. Three months from now, when someone asks "why does the dashboard scroll differently from the tables?", the answer is in specs/008-table-infinite-scroll/research.md, decision R-002.

The numbers

Across our project, spec-driven development with AI produced:

Every one of those 190 tasks was derived from a specification, planned with documented rationale, and executed with explicit dependencies. Not a single feature started with "just write the code."

Trust, but verify

We're not saying AI writes perfect code. It doesn't. During this project we caught edge cases, fixed styling inconsistencies, and debugged issues that the AI missed. But the spec-driven approach changed when we caught those problems.

Instead of discovering architectural mistakes after 500 lines of code, the specification and planning phases surface them before any code exists. Instead of untangling spaghetti from a freeform AI session, every change maps to a task that maps to a user story that maps to a requirement.

The AI is the engine. The specification is the steering wheel.

What's next

Spec-kit is open source and designed to work with any AI coding assistant, Claude Code, Cursor, Copilot, or whatever comes next. We'll keep using it and sharing what we learn along the way.

If you're building software with AI and finding that the output is unpredictable, hard to maintain, or doesn't match what you actually needed, the problem probably isn't the AI. It's the input.

Give the AI a specification, not a wish.

Other tools in this space

Spec-driven development is picking up momentum. If you're exploring the approach, here are the tools worth looking at:

spec-kit: The one we used. GitHub's open-source toolkit that provides templates, a CLI, and prompts to move work through specify → plan → tasks → implement. Agent-agnostic, works with Claude Code, Copilot, Gemini, and others.

Kiro: An AI IDE by AWS (VS Code fork) with spec-driven development built in. You describe requirements in natural language, Kiro generates user stories, technical design docs, and implementation tasks. Great for teams that want the workflow embedded in the editor itself.

Tessl: An agent enablement platform with a CLI that doubles as an MCP server. Its registry indexes 1,000+ reusable skills and docs for 10,000+ packages, keeping agent context version-matched to your dependencies. Focuses on making coding agents more effective through structured, versioned context.

OpenSpec: A lightweight, fluid alternative that works with 20+ AI assistants. Uses an action-based workflow (proposal → specs → design → tasks → implement) with no rigid phase gates, you can update any artifact at any time.

The tooling is still young, but the pattern is clear: the teams that give AI structured input will ship faster and more reliably than those prompting from scratch.

But how do you discover data products when stakeholders themselves might struggle to articulate what they need? How do you gather requirements and design solutions when everyone has a different mental model of how things work?

This is where EventStorming and Context Mapping—two powerful tools from the Domain-Driven Design community—become invaluable for data product managers.

The Discovery and Requirements Challenge

Many stakeholders are accustomed to treating data as a commodity, something they simply request when needed. They may not have the vocabulary or mental models to think about data products strategically. As a data product manager, you need tools that bridge this gap and enable collaborative discovery and requirements gathering.

These are the key components in order to run effective sessions in which you manage to uncover the needs of your consumers:

• Collaboration across diverse teams and functions

• Deep understanding of how the organization actually operates

• A shared language to discuss complex processes and systems

• Frameworks that help stakeholders articulate their needs

Enter EventStorming and Context Mapping

While EventStorming and Context Mapping are distinct techniques, they share powerful principles that make them effective for data product discovery and design:

Bring the right people together. Not managers or directors, but the people who do the actual work: those who feel the pain of inefficient processes and possess deep operational knowledge. These are your domain experts, even if they don't carry that title.

Create a collaborative space. Whether it's a physical whiteboard or a virtual canvas, you need a shared modeling space where everyone can contribute freely. The key is psychological safety: there are no wrong answers, and all perspectives are valued.

Build shared understanding. By visualizing processes, systems, and interactions together, teams develop a common language and mental model that transcends organizational silos.

Context Mapping in Action: Displacing Legacy and Discovering Future State

We recently worked with a global retail company headquartered in the Netherlands undergoing a massive transformation: implementing a new data model to support their product lifecycle management. Dozens of systems across their IT landscape consumed data from the old model, each with custom point-to-point integrations built at different times using different technologies.

The challenge was about understanding a complex web of dependencies, many of which existed only in the minds of developers (some of whom had left the company). Knowledge gaps were everywhere.

Context Mapping became our guide.

We ran sessions with each team managing individual downstream systems, bringing together:

• End users who understood how the tools were actually used

• Developers who had worked on the integrations

• Anyone with contextual knowledge about edge cases and workarounds

Together, we mapped the current landscape:

• How each integration was built and how it works today

• Relationships between systems: upstream versus downstream, who conforms to whose standards, who introduces transformations, and where they happen

• Boundaries between different contexts: what happens when something goes wrong? Who notices the problem, and who's responsible for fixing it?

But Context Mapping isn't just about documenting the present. It helped us envision the future state: how would the introduction of the new product data model reshape this landscape? Who would be responsible for implementing each piece of the solution?

The sessions created something invaluable: shared ownership. Everyone who would play a role in the migration gained buy-in on the solution. We emerged with a clear understanding of implications, impacts, and responsibilities for successful adoption.

EventStorming in Action: Bringing the Bottleneck into the Picture

Within the same program, we hit a roadblock. The design of the new data model struggled to handle a specific edge case in the product lifecycle—an exception to the happy path that threatened to derail implementation.

This is precisely where EventStorming shines.

Same principle: bring in people who do the operational work day-to-day. But rather than focusing on systems and bounded contexts, start with a seed event to kick-start the conversation. No constraints for the participants in terms of time and space, just a huge whiteboard to build collaboratively a timeline of events.

When disagreements emerge about what happens between events, or the correct sequencing, that's your signal as facilitator to dig deeper:

• What are the policies that govern these transitions?

• Which actors and systems are involved?

• What can happen in parallel and what cannot?

Once we had business experts and engineers in the same room, it was easy to construct an accurate timeline of events that allowed everyone to acknowledge the critical gap in the design of the data model. EventStorming helped us seeing the big picture first, and then modeling the chaos, with the support of a standardized grammar to ensure that everybody was using the same language.

The process also served another crucial purpose: the engineers who would implement the solution gained both technical and business knowledge about the purpose and processes behind what they were building. This deep understanding would prove invaluable during development.

Why These Tools Matter for Data Product Managers

As data product managers, our success hinges on our ability to:

• Build empathy with users and understand their Job-To-Be-Done

• Identify bottlenecks and pain points in existing processes

• Unite the team around the "why" behind what we're building

• Deliver solutions that create real business value

EventStorming and Context Mapping are strategic tools that give you structured, collaborative frameworks to achieve all of this. They transform discovery from a vague, frustrating exercise into a concrete, energizing process.

Team members will enter the room as strangers (or in the worst cases, adversaries!) and leave with a shared understanding of what needs to be done next.

Getting Started: Your Path to Collaborative Discovery

Ready to transform your discovery process? Here's how to begin:

1. Start small. Pick one complex process or migration that would benefit from collaborative mapping. Use it as a pilot to demonstrate value.

2. Invest in facilitation skills. These workshops require skilled facilitation to create psychological safety and guide productive conversations. Consider bringing in experienced facilitators for your first sessions.

3. Focus on the right participants. Remember: doers over observers.

4. Embrace messiness. The most valuable insights often emerge from disagreements and confusion—these are signs you're uncovering hidden complexity that needed to be surfaced.

5. Document, but don't over-formalize. The real value is in the shared understanding built during the session, not just the artifacts produced.

Discovery doesn't have to be a shot in the dark. With the right collaborative tools and facilitation, you can turn it into your competitive advantage.

Want to go deeper? We're launching DataChef Academy, where you'll learn how to run effective EventStorming and Context Mapping workshops for data products. Join our waitlist to be the first to know when these hands-on workshops become available.

This isn't entirely wrong: saying no is part of the job, of course. But framing it as the defining skill of product management is dangerous. Worse, it creates a culture where product managers optimize for the wrong outcomes: process compliance, roadmap rigidity, and risk aversion.

Why saying "no" feels like a superpower

Let's be honest: saying no is actually easy. You point at a roadmap. You cite committed priorities. You deflect by adding stuff to the backlog. Done. It's a defensive move that requires little courage and no creativity. Why do I think it’s easy? For at least a couple of reasons.

You’re protected by the process. Especially in organizations that operate with rigid methodologies and strong commitments to roadmaps and planning cycles. When you have a well-defined plan and established priorities, declining new requests becomes a routine exercise that is actually encouraged by organizational principles.

It provides immediate social rewards. Your engineering team appreciates that you're shielding them from constant context switching and scope creep. Your manager values that you're staying disciplined, following the process, and delivering what you had committed to deliver. There's a pleasant sense of being the responsible adult in the room, protecting everyone from chaos.

But here's the uncomfortable truth: there’s nothing special about saying “no”. In most product management careers, there will be far more instances of saying no than yes. The opportunities you don't pursue will always outnumber the ones you do.

The PM who always says “no” is perceived as an obstacle

Worse, encouraging PMs to default to "no" enables a dangerous pattern: product managers who reject everything outside the plan train stakeholders to work around them.

When business stakeholders or customers present something urgent, they're not trying to derail your carefully crafted plans. They're bringing you a problem they believe is important. In many cases, they're right. They're closest to the customer pain, the market dynamics, the competitive threats. They have information you might not have.

Great product managers recognize this and treat it as valuable input, not an annoyance to be deflected. They ask questions: Why is this urgent now? What happens if we don't address this? Who is affected and how severely? What outcome are you trying to achieve? After asking questions, they ponder and offer options.

When product managers consistently demonstrate the willingness to find a way forward, they build tremendous stakeholder confidence. Stakeholders learn that this person is thinking first and foremost about solving the right problems, not blinded by process for the sake of process. They will see them as a partner in achieving business outcomes.

Conversely, product managers who shut down conversations about unplanned work will be seen as obstacles. Stakeholders lose patience after hearing for the third time that their customer pain point is "in the backlog." They escalate to executives, who then mandate the work anyway. In the worst cases, stakeholders might even secure budget to build shadow solutions outside your product. In the process, you lost the chance to own the problem space. Worse, you lost credibility and authority, neither of which is easily regained.

What actually defines great product leaders

Great product management is about understanding the "why" before the "what" and "how." A product manager's core responsibility is to own the “why”, i.e., to maintain focus on business outcomes and end-user needs.

This means that the real differentiator of their success is not how often they say no, but which opportunities they say yes to and the impact they will generate on the business.

The challenge—and the art—lies in spotting the right opportunities that make you go: "Yes, we're going to change our roadmap. We've found something that is actually more important and more impactful to do."

This shouldn't happen every day. If you're constantly changing direction, you create chaos and undermine trust. But when the right opportunity presents itself, great product leaders (at any level: ICs, directors, C-level) have the instinct to recognize it and the courage to act on it.

The anatomy of engineering a "yes"

So, what does it mean to engineer a “yes”? At DataChef, we believe it involves three things that are part of the core skill set of a good product manager.

Rapid validation. When a potentially important opportunity surfaces, skilled product managers quickly validate whether it's worth pursuing. They have a strong instinct that helps them uncover the underlying problem, even when the pain point is poorly articulated or buried in raw feedback. They spot patterns and connect the dots, tying new requests to problems that surfaced earlier or recognizing similarities with already-prioritized work.

Creative solution design. Here's where the engineering comes in. Saying yes doesn't mean accepting the proposed solution at face value or dropping everything else. It means coming up with options to address the high-impact problem while managing constraints. For example, descoping less critical work, negotiating scope to deliver 80% of the value with 20% of the effort, or identifying overlaps with existing priorities that address the same need.

Stakeholder alignment without creating uncertainty. The hardest part of engineering a “yes” is doing so without creating chaos in the team or losing stakeholder confidence. This requires clear communication about why this opportunity matters more than what was planned and transparency about what's being deprioritized.

It’s on leaders to build a culture of options

If you’re a Director of Product, or a Chief Product Officer, here’s my advice to you. Companies get the product management they incentivize. If you reward product managers for sticking to the plan, minimizing stakeholder requests, and keeping teams busy with predetermined work, you'll get gatekeepers and coordinators. In many organizations, especially in Europe, that is the status quo.

If instead you reward product managers for delivering measurable business outcomes, solving high-impact customer problems, and building strong stakeholder relationships, you'll get product managers who think creatively and take calculated risks while driving real value.

The real trap is confusing process compliance with product excellence. Leaders should be careful about the metrics they pick to measure success and the stories they promote as virtuous examples.

And when performance review time comes, remember: it takes courage to engineer a “yes”. The product managers who get invited to strategic conversations, the ones who build loyal followings among stakeholders and teams, are the ones who figure out how to say yes to the right things.

So the next time you hear someone say "the product manager's superpower is saying no," push back. Because at the end of the day, nobody remembers the product manager who was really good at saying no. They remember the product manager who found a way to solve the most important problems, even when it wasn't easy.

We are now entering a new stage of analytics. We are shifting away from static dashboards and moving toward conversational intelligence: unlocking an organization's institutional memory through natural language.

The question is no longer if you should adopt Generative BI, but how. Do you lean into the Microsoft ecosystem with Copilot, or does your context justify a custom architecture running on your own terms?

Copilot vs Custom: Quick Comparison

Note: A custom open-source stack typically combines components like LangChain/LlamaIndex for orchestration, open-source models (Llama/Mistral), a vector store (Elasticsearch/pgvector/Pinecone), and your existing warehouse/lake, all deployed within your current infrastructure (e.g., Kubernetes, VPC, existing IAM).

The Promise: Power BI with Copilot

Microsoft has integrated Copilot into Power BI, and for many organizations this is the most direct path into Generative BI. The vision is to remove technical barriers, allowing decision-makers to access data as easily as sending a message.

Imagine a sales director who needs to prepare for a quarterly review. Instead of filtering through slicers on a complex report, they type:

“Show me sales performance for Q3 compared with Q2. Break it down by region and point out the three weakest products.”

Copilot prepares a visual and a short explanation of why performance changed.

They follow up with:

"Why did our gross margin drop in October?"

Copilot scans the dataset and explains:

Gross margin fell by four percent because supply chain costs for Product Line X rose by fifteen percent.

This is the promise: simple questions, instant answers, automatic visuals. No need to open a ticket or wait for the next sprint.

Where Copilot Shines

It is important to be clear where Microsoft has built a real advantage. If your data strategy is centered on Azure, Copilot offers benefits that are hard to match with a purely custom stack:

• Seamless integration: It lives where your users already work: inside Power BI, Teams, and Excel. That matters for adoption.

• Identity & Governance: It leverages Entra ID (formerly Azure AD) for robust Role-Based Access Control (RBAC). If a user is restricted by row-level security, Copilot will not see that data either. You get consistent access rules from reports down into conversational queries.

• Auditability: With Microsoft Purview integration, you get built-in lineage, cataloging, and sensitivity labeling. This goes beyond just security. Analysts can trace where an answer came from, compliance teams can reason about risk, and executives can trust that numbers are not appearing from nowhere.

The Decision Matrix: Choosing Your Path

While Copilot is powerful, it is not a universal answer. Implementing Generative BI via the standard Microsoft route comes with specific infrastructure requirements and constraints that may not fit every agile or cost-conscious business.

Here is a framework to help you decide which approach fits your organization.

When Copilot + Power BI is the Right Answer

Staying with the official Microsoft platform is often the best choice if:

1. You are a "Microsoft Shop": Your organization already uses Entra ID, Purview, and Power BI Premium or Fabric capacity. Your BI teams are comfortable with Power BI as the main semantic layer.

2. You have solid semantic models: You have well-maintained semantic models. Copilot relies heavily on the quality of the underlying semantic layer; it struggles with messy or unstructured data.

3. Governance and auditability are top priorities: You want audit logs, sensitivity labels, and lineage in a form that security and compliance already understand. You would rather inherit this from Entra + Purview than rebuild it.

4. You accept the Fabric/Premium commitment: You are willing to pay the "Fabric Tax" (committing to Fabric or Premium capacities) for the ease of a fully managed service.

When a Custom Open-Source Solution Makes Sense

A custom Generative BI layer, built with open-source components and your own warehouse, becomes compelling when:

1. You have strict data-residency or “offline” needs: In industries like defense, healthcare, or finance, sending data to a public cloud-hosted LLMs is not acceptable. You may need to run open-source models like Llama 3 or Mistral on-premise or in a private VPC where data never leaves your control.

2. You care deeply about cost control and routing: Capacity-based billing for Fabric means you pay whether Copilot runs one query or one million. With a custom solution, you have control over model routing, allowing you to leverage low-cost models for most use cases and reserve more expensive reasoning models for complex tasks.

3. Your “truth” lives in the warehouse, not only in Power BI: You want your conversational interface to talk directly to gold tables in Databricks, Snowflake, or other warehouses, and to combine that with unstructured documents via RAG and agents. Copilot is optimized for Power BI semantic models; a custom agent can be designed around your broader data estate.

4. You want vendor flexibility: You want the freedom to swap the "brain" of your operation without changing your entire platform. If a new, faster model is released tomorrow, you want to plug it in immediately without waiting for a vendor update.

The Friction: Why Copilot Isn’t for Everyone

While Copilot’s capabilities are undeniably impressive, adopting it via the standard route often comes with substantial prerequisites and constraints — a reality that can be a barrier for agile or cost-sensitive organizations.

• “Capacity” costs required: To enable Copilot in Power BI (or the larger Microsoft Fabric ecosystem), your workspace must be hosted on a paid capacity — either Fabric capacity (F-SKU) or Power BI Premium capacity (P1 or higher). It’s not sufficient to only have a free or trial license. Microsoft Learn

• Fixed overhead, even if usage is low: Because capacity-based billing applies regardless of actual usage, you may incur infrastructure costs even if few Copilot queries are made. This can be a heavy commitment — especially for organizations with sporadic or unpredictable usage.

• Ecosystem lock-in: Copilot is deeply tied to Microsoft infrastructure, APIs, and model choices. That is a benefit if you are all‑in on the stack. It is a constraint if you want to experiment with open-source models, alternative warehouses, or a multi‑cloud architecture.

• Compliance & data-residency restrictions: For industries with strict regulatory requirements (finance, healthcare, etc.), using Copilot via cloud-based LLMs can raise concerns about data residency, governance, and compliance. While there are regional and tenant-level settings in Fabric, they require careful configuration. Microsoft Learn

The DataChef Approach: Co-Designing Your Architecture

At DataChef, we do not sell a boxed “GenBI product” that competes with Microsoft. We help you design and implement the architecture that fits your constraints, which often includes Copilot.

For some clients, that means tightening their Power BI models, access rules, and Purview setup so Copilot can actually be trusted. For others, it means designing a custom “institutional memory” engine that runs in their own environment, side‑by‑side with existing BI.

Here are a few patterns we tend to design for:

1. Specialized Security & "Air-Gapped" Intelligence

When zero data leakage is non-negotiable:

• Deploy open-source models fully within your private cloud or on‑prem environment.

• Fine‑tune models on your own jargon and metrics definitions so answers reflect how your organization speaks and reasons.

• Keep all prompts, logs, and embeddings inside your own perimeter.

2. Flexible Intelligence Through Model Routing

When cost and performance both matter:

• Route simple, high‑volume queries to small, local models.

• Reserve top‑tier hosted models (e.g. Claude, GPT‑4 class) for complex reasoning, planning, or edge cases.

• Evaluate and swap models as the landscape evolves, without changing your front‑end experience.

3. Governance by Design

When governance must span both Copilot and custom stacks:

• Integrate agents with your existing identity and access management (for example Entra or other IAM), so the same rules apply everywhere.

• Log questions, answers, and underlying queries in your environment so you can:

  • See what people are actually asking.

  • Identify gaps in your semantic models or warehouse.

  • Improve definitions and data products over time.

Conclusion: Partnering for the Future

The shift to conversational BI is underway. The exact tool you use (Copilot, a custom agent, or both) matters less than whether you have:

• A clear view of your current BI and data platform.

• An honest assessment of your governance and residency constraints.

• A cost model that fits how you actually plan to use Generative BI.

For many Microsoft‑centric organizations, the right move is to start with Copilot in Power BI, provided the semantic models and governance are in good shape. As soon as you hit residency, multi‑cloud, or advanced routing needs, it becomes worth designing a custom Generative BI lane alongside it.

How we help If you are wrestling with this choice, a good first step is to take one or two concrete use cases - rather than your entire BI estate - and ask:

• Can Copilot, on top of our current models and governance, serve this well?

• Where do residency, cost, or multi‑platform needs push us beyond what Copilot can reasonably cover?

This is the kind of work we do with clients: we map your current BI setup, security requirements, and cost constraints, then co‑design the smallest viable next step - whether that is getting more value out of Copilot, adding a focused custom GenBI slice, or combining both into a coherent architecture.

Most conversations frame this as a choice between data lake and data mesh. Vendors, internal teams, and reference architectures encourage you to pick a side.

The more useful question is:

Does your organization have the structure, skills, and governance to operate the architecture you are buying into for the next 3-5 years?

If the answer is no, the right pattern on paper will not help in practice. You are making an operating‑model choice, not just a technology choice. If the model does not fit how your organization works, you lock in extra cost and rework, slower decisions, and a credibility problem for the data team.

If the match is roughly right, the effect is the opposite. You shorten time‑to‑change for important decisions, reduce shadow solutions, and build an architecture that supports the ambitions of the organization instead of fighting them.

This article is about that match. We will not argue for lake or mesh as abstract patterns. Instead, we will help you:

• Judge whether your organization is genuinely ready for a more decentralized model.

• See what you can already apply from data mesh thinking on top of the central lake or warehouse you run today.

By the end, you should be able to answer a practical question: what level of decentralization can we operate safely now, and what can we grow into without breaking the business?

What a Centralized Platform Is (and When It Fits)

In most organizations, the “data lake vs data mesh” decision is really a question of how centralized your platform is. A centralized platform — typically your data lake or warehouse — means one team owns the shared environment. Data lands and is modeled in one place, under one set of standards. Most meaningful changes flow through a single backlog and roadmap.

This is often the right starting point. You are moving from chaos to order. Data ownership in domains is still fuzzy. Your main pain is fragmentation and duplication, not yet a central bottleneck. In that context, a central platform behaves like a good monolith: one place to reason about data, one set of contracts, one team accountable when something breaks.

The trade‑off is clear. You concentrate capability, governance, and decision‑making in one place, and accept that many new use cases will queue behind the same team. At limited scale, this is usually a good deal. It lets you enforce standards, simplify tooling, and ship visible wins, especially if you are coming from scattered spreadsheets and side systems.

As volume, teams, and use cases grow, the same design can quietly become the throttle for change rather than the enabler. The central team’s backlog starts to define how quickly the business can move.

The question is not “is centralization bad?”, but “given how we actually work today, are we still at the scale where a single team can safely sit in the middle of everything?”

What a Decentralized Platform Is (and When It Fits)

At the other end of the spectrum is a decentralized platform — data mesh in practice. Instead of one central team owning most pipelines and models, stable domain teams own their data products end to end. They design the models, run the pipelines, and are accountable for quality and contracts. A central group still exists, but focuses on platform, guardrails, and enablement.

This model makes sense once your main problem is no longer “we have nothing consistent,” but “the central team is slowing everyone down.” Domains already move fast in their own products and services. They want similar control over the data that represents their part of the business, and they have enough engineering and analytics capability to carry that responsibility.

In that situation, a single central backlog has become the limit for how quickly the business can change. A decentralized model changes that constraint: you keep a strong shared platform, but move more decisions, design, and implementation into the domains that know the business best.

Moving to a decentralized model is like moving from a monolith to well‑designed services. You keep a shared foundation, but push ownership and change closer to where domain knowledge lives. When it fits, this gives you faster decisions, tighter alignment between data and reality, and fewer shadow solutions, because teams no longer need to route around a central bottleneck to get work done.

Here too, the question is not “are we doing data mesh?”, but “do we have the structure, skills, and trust to let domains own real data products on top of a shared platform, without collapsing into chaos?”

Antipatterns: When Each Model Fails in Practice

Centralized and decentralized platforms both work when they match how the organization actually operates. Even when the technology is sound, the wrong match with your organization creates failure patterns that look very similar across companies.

Forcing Data Mesh into an Unready Organization

You are likely forcing mesh too early if you recognize several of these signals:

• Org shape vs. design do not match

  You still run mainly as functions or projects (BI team, central data, “the business”), but your target diagram is full of tidy domains and data products. In practice, work and budgets still route through the old structure.

• Domains “own” data only on paper

  You have named data product owners, but they have no time, no engineers, and no real mandate. At quarter‑end, they still send SQL snippets or requests back to the central team to “fix the numbers.”

• Everyone is rebuilding the basics

  Each domain is trying to figure out its own ingestion, testing, and monitoring. There is no clear golden path from the platform. Teams ask each other “how did you solve X?” because nothing is shared.

• Platform and practices are still fluid

  Tooling, environments, and standards are changing under people’s feet. New ways of doing things keep appearing before the old ones have settled. The idea of “letting every domain choose” feels like multiplying today’s instability.

• Big‑bang thinking instead of small, contained pilots

  The transformation is framed as “we are doing data mesh” across the whole estate. There is no small, end‑to‑end domain where you can point and say: “this is working, here is how, here is what we learned.”

If several of these resonate, the problem is not that mesh is a bad idea. It is that you are asking the organization to run an operating model it does not yet have the structure, skills, or platform to support.

Staying Centralized When the Lake Team Is the Bottleneck

On the other side, you may be clinging to a central lake model after it has clearly outlived its scale. Common signals:

• The central backlog is the throttle for business change

  Most meaningful analytics or data changes must pass through one team. Lead times are counted in weeks or months. Leaders talk about “waiting for the data team” as a standard part of delivery.

• Shadow data and parallel stacks keep popping up

  Teams run their own extracts, spreadsheets, and side databases “just for now.” Some units experiment with their own BI tools or cloud accounts because they see no other way to move.

• Trust in “official” data is eroding

  People compare dashboards from different sources in meetings. Arguments start with “that’s not what my numbers say.” The central platform is still branded as “single source of truth,” but behaviour says otherwise.

• The platform is seen as something done to domains

  Business teams feel they have little say in schemas or priorities and experience the central team as a gate, not a partner. Even when they use the official path, it feels misaligned with their reality, so they increasingly disengage from the central solution.

• Central teams are fighting symptoms, not causes

  Most effort goes into patching fragile pipelines and firefighting incidents in unfamiliar domains. Root‑cause fixes require deep business context that sits with local teams, but ownership has never really moved there.

If these signals are familiar, your issue is no longer “do we need more standards?” or “do we need a better tool?”. The central lake itself is now the limit for how fast you can change. That is usually the moment to start pushing ownership and accountability closer to domains, on top of a strong shared platform, instead of adding yet another layer of process around the same bottleneck.

A Readiness Lens for Your Organization

Moving from a centralized approach to a more decentralized one is not a simple on–off switch. It needs a deliberate roadmap and preparation.

The questions below give you four key lenses to assess whether your organization is ready for that step. If your answer is “no” for one of them, the “Action” under that lens shows where to invest next so you move closer to a decentralized model you can actually run.

Ownership

Ask:

• Have you already identified your domains?

• Do domains already own services, APIs, or key analytics for their area?

• Can you name accountable data owners with enough time and mandate, not just a title?

If ownership is diffuse and always rolls back to “the data team”, you are not ready for a broad mesh.

Action: Start by clarifying ownership inside a central lake model.

Platform Maturity

Ask:

• Do you have a shared platform for cross-cutting concerns, especially security and observability?

• Or would each domain have to assemble its own solutions in order to build data products?

If each domain would be rebuilding basics on a moving foundation, decentralization just multiplies instability.

Action: Stabilize the central platform and define a “golden path” before pushing ownership out.

Governance and Trust

When we talk about governance here, we mean the practical rules for how data is created, changed, and used across the organization. That includes data governance topics like definitions, quality, access, and lineage, and also the decision rights around who can approve changes, who owns which datasets, and how exceptions are handled. In other words: the minimal set of shared rules and decision paths that keep data trustworthy and compliant.

Ask:

• Are standards followed mainly because they are useful, or because a committee enforces them?

• Can you define a small, clear set of rules and trust domains to operate within them?

If every change requires escalation and exception handling, added autonomy will create more variance, not more value.

You need governance that works as guardrails, not as a gate for every decision.

Action: Make governance work as guardrails with pre-approved paths that teams can adopt without asking permission.

People and Skills

Ask:

• Do domain teams have engineers or analytics engineers who can own pipelines end to end?

• Do you have an enablement team that helps domains adopt shared practices and technology — coaching them, providing templates, and pairing on real work?

If these skills sit only in one central team, you will either overload it or hand responsibility to people who cannot carry it.

Action: Invest in skills and enablement before shifting significant lifecycle ownership into domains.

It’s Not a Binary Choice: Borrow the Best Ideas

You do not have to pick a side and live with it forever. Most healthy organizations end up with a central platform and varying degrees of decentralized ownership on top.

Apply Mesh Ideas on a Central Platform

You do not need to re‑architect everything to benefit from data mesh thinking. Many of the ideas work well on top of a central lake or warehouse you already run.

In practice, the pattern that works is:

• Keep the platform centralized.

  One place to manage infrastructure, security, governance, and the golden path.

• Decentralize data products.

  Domains own the tables, models, and APIs that represent their part of the business, on top of that shared platform.

On that foundation, apply “mesh” ideas inside your lake or warehouse:

• Treat important datasets as products: give them clear owners, roadmaps, and SLAs.

• Make contracts explicit: schemas, refresh cadence, and rules for breaking changes.

• Bring domain experts into design and prioritization, instead of letting a central team guess what they need.

You get leverage from a shared foundation, while pushing accountability closer to where decisions and knowledge live.

Sequence the Journey

You do not need, and likely do not want, a big‑bang transition.

A pragmatic sequence:

1. Stabilize a central lake or warehouse.

   Get reliability, governance, and observability to an acceptable baseline.

2. Introduce product thinking and ownership.

   Name owners for key domains and define clear contracts.

3. Gradually decentralize where it adds leverage.

   Let mature domains take on more of the lifecycle, on top of the shared platform.

At each step, ask a simple question: can your organization operate this level of decentralization without creating chaos?

You Don’t Have to Make This Choice Alone

At DataChef, we have helped organizations design and evolve data platforms across warehouses, lakes, lakehouses and meshes. The patterns that work in practice are always the ones that start from the business.

The shift from a centralized to a more decentralized model does not happen overnight. You need a clear view of your structure, skills, and governance, and a roadmap that links operating model and technology to outcomes. That is why we look beyond tools and architectures and focus equally on organisational design and team boundaries, using approaches like Team Topologies

If you are considering a move towards data mesh or rethinking your central platform, we would be happy to help you assess where you are today and design a path you can truly own.

A quiet shift is now underway. It is changing how leaders work with data, how quickly they make decisions, and how often they validate their intuition. That shift is Generative BI.

Despite the futuristic name, it is not a tool. It is a capability.

From Dashboards to Conversation

The simplest way to understand Generative BI is to compare two worlds. Imagine you are reviewing last month’s performance and notice that margin dipped in one region. In the old world, you would email your BI lead, ask for a breakdown by product and channel, and wait a few days for a new report. By the time it arrives, the team has already moved on to other priorities. In the new world, you ask: “Why did margin fall in Region North last month?” You immediately see the answer broken down by product line. You follow up with “Show me the top three customers that changed the most compared to the previous quarter” and “How much of this is due to discounts?” You get to a decision in minutes, while the topic is still fresh in your next leadership meeting.

That is Generative BI: a conversational layer on top of the data and models you already have, designed to unlock faster thinking and faster decisions. This does not replace analytics. Dashboards remain essential for recurring KPIs and long‑term performance tracking. What changes is the interface. Executives and domain experts no longer need to file tickets for every ad‑hoc question. They can simply ask and explore.

Why Executives Should Care

The value of Generative BI has little to do with AI as a buzzword. Its impact is felt in the pace and quality of decision‑making.

Faster, More Accountable Decisions

Instead of waiting for someone to build or modify a report, leaders can get to the why in real time, directly in conversation with their data. This is powerful in areas where timing compounds results: pricing, campaign optimization, supply chain adjustments, store or channel operations. Being able to ask “what changed?” at the moment you notice something creates a fundamentally different decision cadence.

At the same time, when the path from question to answer is transparent – which data was used, which definitions were applied, which assumptions were made – it becomes easier to challenge, refine and document decisions. Generative BI does not just speed up decision‑making. It makes the link between narrative, numbers and ownership much clearer.

A Better Return on Data Investments

Most organizations have already invested heavily in warehouses, modeling and BI tools. Generative BI does not replace that work. It helps people finally use more of what they already have, by making it far easier to get from question to insight.

Relief for the BI Backlog

BI teams are excellent at producing repeatable dashboards. What they cannot cover is the long tail of everyday questions that never get prioritized, simply because nobody has time to build a view for them. Generative BI absorbs that long tail. It answers the quick, exploratory questions that today either never get asked or arrive as “just one more request” for the analytics team.

More Curiosity Across the Organization

When people do not need to open tickets or wait for sprints, they ask more questions. They explore more scenarios. They validate more hypotheses. Many of the best ideas in an organization start with someone noticing a small anomaly and digging deeper. Generative BI makes that investigation immediate, instead of dependent on the next reporting cycle.

The Honest Limitations

Like any powerful capability, Generative BI has real constraints. Being clear about them is the difference between a valuable system and a disappointing pilot.

Your Data Still Needs to Be Right

If definitions are inconsistent or tables do not line up, the system will give you answers quickly, but they may be confidently wrong. Generative BI exposes weak data foundations. It does not repair them.

Nuanced Business Concepts Are Not Obvious to a Model

The system can easily answer structured questions such as trends, comparisons or top‑N performers. It struggles when underlying concepts are ambiguous or live inside people’s heads. Terms like “active customer,” “at‑risk account,” or “adjusted margin” often require shared understanding across teams. Unless those definitions are made explicit, a model cannot apply them reliably.

People Still Need to Trust the Numbers

Executives will always ask where a number came from. A good Generative BI setup must show how an answer was produced: which tables it touched, which metrics it used and what logic it applied. Without that transparency, leaders will treat the system as a toy rather than a decision support tool.

Governance Matters More, Not Less

A conversational interface makes it easy for anyone to ask anything. Without the right guardrails, people may access information they should not see or draw conclusions from partial context. Thoughtful governance, access control and guardrails are what turn Generative BI from a risk into a trusted capability.

Put simply: Generative BI will not magically fix weak data foundations or misaligned definitions. It amplifies whatever you already have, good or bad.

Making Generative BI Work in the Real World

The hard part of Generative BI is rarely the interface. Most demos look magical. The real challenge is making it work reliably inside the messy, nuanced reality of a business.

This is where DataChef focuses: turning promising generative BI demos into a durable capability on top of your real data.

From Demo to Capability

Real data is messy. Business rules clash. Metrics drift over time. DataChef works with both domain leaders and your data/BI teams to get a complete view: not just how the data is modeled, but how the business actually makes decisions. We use focused workshops (for example, event storming, value stream mapping, and Wardley mapping) with a multi‑disciplinary group of client stakeholders. Together, we identify the critical decisions you want to support in conversation, then shape the underlying data and definitions around those questions. The goal is not a one‑off proof of concept, but a capability that executives can return to every day.

Designing a Safe Environment for Exploration

We work with you to decide which domains, metrics and tables should be accessible first. Together, we make explicit which sources are in scope, which definitions are authoritative, and which caveats must always be surfaced. On top of that, we put access controls and audit in place. The result is a “safe playground” where executives and domain experts can explore without risking misinterpretation or exposure of sensitive data.

An Iterative, Low-Risk Approach

You do not need a big‑bang transformation. We prefer a thin vertical slice (or tracer bullet) approach: prove the value end‑to‑end as quickly as possible by taking a narrow slice, for example a specific sub‑domain. That slice includes everything from the data model and definitions to the conversational experience and governance. This way, you get feedback on every part of the solution early, and you can expand based on real usage and learning rather than on abstract requirements.

What This Means for Your Organization

Generative BI is not the end of dashboards, and it is not a replacement for your analytics team. It is a new capability that changes how quickly leaders can think, explore ideas and act with confidence.

For organizations ready to embrace it, the shift is profound. It allows decision‑makers to move at the speed of conversation, supported by data they already own.

If you are curious what this could look like for your company, DataChef helps organizations design that first step: a secure, trusted conversational layer that unlocks meaningful, fast insights for leaders.

At some point we realized something simple. We spend a lot of time welcoming people, but the work behind that welcome is heavy and invisible. We wanted to keep the warm feeling while removing the heavy lifting around it. So we built our own onboarding workflow in n8n, and the difference was bigger than we expected. Today the entire onboarding process runs in seconds. It feels consistent, personal, and honestly a little magical.

Screenshot: The full onboarding workflow in n8n

1. A tiny Slack form that starts everything

It all begins when someone from PeopleOps fills in a short Slack form. It is simple and quick, containing only the basic details like name, position, and start date. Behind this small step, the workflow wakes up and starts preparing everything in the background. It feels like ringing a small bell in the kitchen and watching everything fall into place without any stress. This one form replaces hours of careful setup and long checklists.

Screenshot: Slack shortcut trigger

2. The new colleague’s digital identity appears like magic

Right after the form is submitted, n8n starts building the new colleague’s digital identity. It checks for an available email address, chooses one, creates it, generates a temporary password, and connects their personal email. Before automation, this took careful typing and double checking. Now the process finishes almost instantly. Watching it happen feels like someone quietly preparing the perfect setup behind the scenes.

Screenshot: Email creation section

3. Their Notion space grows around them

Notion is our central place for documentation and shared knowledge. We wanted new colleagues to arrive and instantly feel that everything was ready for them. The workflow creates a DOP (DataChef Orientation Program) page to guide their first days and a One Team page that stores their basic information. Both pages come filled with the right layout and details. There are no empty pages or forgotten fields. Everything looks prepared with intention and care.

4. A warm welcome message lands in their inbox

Before they even join Slack, the new colleague receives a friendly welcome email. It includes their company email, a temporary password, and simple next steps. This is often the first moment of surprise for them. It shows that the company has already prepared a space, and all they need to do is walk in.

5. n8n patiently waits for them to join Slack

This part has a calm and almost playful feeling. The workflow does not rush or create channels before the person actually joins Slack. Instead, it waits and checks from time to time whether the new account has appeared through SSO. When the new colleague finally signs in, n8n continues the process instantly, almost as if saying, “Welcome. Let us continue.”

6. Their Slack channels appear fully ready

Once the new colleague is active in Slack, things start to appear around them. Their DOP channel is created, a progress review channel appears, clear instructions are posted, reminders are sent, and the right people are invited. Before automation, these steps were done one by one and not always on time. Now everything is ready exactly when it should be. This gives the team time to focus on the human side of onboarding, sending greetings and welcoming the new colleague into the kitchen.

Screenshot: Slack channel automation

7. Engineers get GitHub access without delay

For engineers, the workflow includes one more useful step. It asks them for their GitHub username directly inside Slack. They respond, and n8n updates our GitHub Terraform repo and gives them the correct access. There are no delays and no forgotten requests. It is fast and reliable.

Screenshot: GitHub automation

8. PeopleOps finishes with a human touch

Even with all this automation, we still keep an important human moment at the end. PeopleOps receives a review form that covers hardware, internal tools, and a few things that still need personal attention. This keeps the balance right. Automation takes care of the predictable work, and people focus on the meaningful parts of onboarding.

The result feels peaceful, simple, and very DataChef!

What used to be a long list of steps is now a calm and clear experience. New colleagues feel supported from the first moment. The team feels more relaxed. PeopleOps has time for the warm, personal parts of onboarding instead of repeating small tasks. Onboarding used to be something we had to manage carefully. Now it is something we enjoy watching. It feels like the company itself reaches out and says, “Welcome to the kitchen 🧑‍🍳, everything is ready for you!”

Leaders with reductionist tendencies are always looking for a formula, especially in the world of technology teams. Some of them may have thought they found one in the rigid, comforting set of options in the 1st book published by Skelton and Pais — a grammar, indeed, to label teams and their interactions.

Team Topologies practitioners use those team types and interaction modes both in a descriptive way (to depict an organization’s current state) and in a prescriptive way (to design its intended state). But if those reductionist leaders had been in the room with us at Fast Flow Conference, they would have realized that such patterns were never meant to be used in isolation.

Over the years, the Team Topologies community has expanded its toolkit. It now includes old and new methodologies, critical to develop a much deeper understanding of the dynamics of how organizations evolve over time. I call this toolkit the Language of Flow. Team types and interaction modes constitute the grammar of that language.

Over two days of exciting talks, I heard success stories highlighting the importance of EventStorming, Wardley mapping, value stream mapping, and systems thinking — a set of collaborative techniques that complement each other by putting humans at the center of the stage. I saw a strong emphasis on the role of enabling teams and on the principle of sharing knowledge across silos. I attended workshops where platform engineers were given discovery tools drawn directly from the best product management practices. And I saw two key figures in the Team Topologies world, Matthew Skelton and Joao Rosa, delivering talks that broadened everyone’s perspective by zooming out from the narrow focus on engineering teams to discuss strategic topics such as budgeting, economies of scale vs economies of ‘empowerment’, compliance, and more.

None of that felt forced or like a hard sell, and that’s precisely because all the speakers share the same Language of Flow, a toolkit that can reshape organizations by thinking human-first. The original patterns of Team Topologies are a core part of that language, acting almost like an orchestrator, but they lose their transformative power when they are introduced in an organization that is not fluent in the Language of Flow.

If you thought applying the grammar rules to your teams was enough to change how your organization works, you've missed the bigger picture. You need to introduce the Language of Flow as a whole.

DataChef has a proven history of being an agent of change in companies such as PostNL, CarNext, Rituals, and Allseas. Reach out if you need to accelerate your journey.

When monitoring applications and infrastructure, businesses usually choose between agent-based and non-agent based monitoring solutions. Some tools, such as Datadog or Splunk, can use agent-based approaches, while both also support non-agent methods. In this article, we’ll use Datadog as an example of agent-based monitoring and Splunk as an example of non-agent-based monitoring, to show the differences in practice. Each approach has its strengths and challenges, depending on needs, environments, and monitoring goals.

Agent-Based Monitoring

Agent-based monitoring means installing a small program (an agent) on each server or host that you want to monitor.

How it works

• An agent runs continuously on the host machine.

• It collects metrics like CPU, memory, disk usage, application logs, and more.

• Data is sent to a central server (like DataDog's cloud) for analysis.

from datadog import initialize, statsd

# Initialize (if sending directly to local agent, defaults are ok)
options = {"statsd_host":"localhost", "statsd_port":8125}
initialize(**options)

# Send a gauge metric (e.g., app.queue.size=5)
statsd.gauge("app.queue.size", 5)

Pros

• Rich Data Collection: Agents can collect detailed performance metrics and logs directly from the machine or container.

• Real-Time Monitoring: Since agents run locally, they can send data almost instantly.

• Custom Checks: You can configure agents to do extra checks, like custom scripts.

• Easy Auto-Discovery: Agents can sometimes auto-detect services and start monitoring them automatically.

Cons

• Deploy and Maintain: Every machine or container needs an agent installed and kept up to date.

• Resource Usage: Agents use a bit of the host’s CPU and memory, though this is usually small.

• Compatibility: Some environments (like highly restricted or legacy systems) may not allow agent installation.

Non-Agent Based Monitoring

Non-agent based monitoring collects data without installing anything on the host. A central system pulls in data, often by receiving logs or metrics through APIs, syslog, or other protocols.

How it works

• Systems send their log files, events, or performance data to a central collector (like Splunk).

• No agents run on the host; configuration often happens on log shippers or using built-in system protocols.

import requests
import json

splunk_url = "<https://splunk-server:8088/services/collector>"
headers = {
    "Authorization": "Splunk YOUR_HEC_TOKEN"
}
data = {
    "event": "metric",  # event type
    "fields": {
        "metric_name:app.queue.size": 5
    }
}

# Send the data to Splunk
requests.post(splunk_url, headers=headers, data=json.dumps(data), verify=False)

Pros

• No Agent Management: Nothing to install or update on the monitored systems.

• Good for Legacy/Restricted Systems: Useful where you cannot install extra software.

• Centralized Control: All settings and updates occur on the collector’s side.

Cons

• Limited Data: Sometimes, only basic metrics or logs are available unless the system supports rich exports.

• Slower or Batch Updates: Data may arrive in batches, so insights may lag behind real-time.

• Harder Customization: Custom health checks or metrics are harder to set up.

Distributed Tracing

What is distributed tracing?

Distributed tracing helps developers follow the journey of a user or API request as it moves through different services in a system. Each step the request takes is called a “span.” Each span gets a span ID (unique identifier).

Where is it used?

• Microservices: When applications have many small services talking to each other.

• Serverless and Cloud-Native Apps: Where requests touch multiple services.

• Debugging Performance Issues: To find bottlenecks in big, complex systems.

How is it used?

• When a request starts, a trace ID and a span ID are created.

• As the request travels, new spans and IDs are made for each new service or step.

• All the spans are grouped together under the single trace ID.

• This lets you see the entire path—the trace—of a request, how long each step takes, and where failures happen.

Span ID

• Every operation in the trace has its own span ID.

• The span ID helps to track and organize all steps within a single trace.

• By analyzing span IDs, you can see how long each segment took and how services relate to each other.

When to use each monitoring solution

• Visualization: DataDog has easy-to-use dashboards to view full traces and performance of each span.

• Friendly for Developers: Many programming languages and frameworks are supported with minimal manual work.

• Built-In Agent Support: DataDog’s agent natively collects distributed traces together with logs and metrics. This makes setup simpler and deeper.

• Real-Time Tracing: The agent sends tracing data in real time, which helps with quick debugging.

• Automatic Context Linking: Traces, logs, and metrics are tied together, making it easier to investigate issues.

• More Control: If you want to have more control over OpenTelemetry abstraction, Splunk can be a better option. Although the OpenTelemetry library is still under development.

By contrast, while Splunk can handle traces through external plugins or integrations, it often requires more manual setup and may not offer real-time or tightly integrated tracing experiences out-of-the-box.

Simple Python example of how distributed tracing can be done in DataDog:

from ddtrace import tracer, patch_all

patch_all()

@tracer.wrap()
def say_hello():
    print("Hello from DataDog tracing!")

if __name__ == "__main__":
    say_hello()

Same example with Splunk:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

otlp_exporter = OTLPSpanExporter(endpoint="<https://ingest.us1.signalfx.com/v2/trace>", headers={
    "X-SF-TOKEN": "your-splunk-access-token"
})

trace.set_tracer_provider(TracerProvider())
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("splunk-example-span"):
    print("Hello from Splunk tracing!")

Conclusion

• Both Datadog and Splunk offer agent-based and agentless options. The choice is less about the tool and more about your monitoring strategy: agent-based for richer, real-time detail, and non-agent for environments where agents aren’t possible.

• For distributed tracing, Datadog offers strong out-of-the-box support, while Splunk emphasizes flexibility and standards like OpenTelemetry.

A web API is (mostly) predictable. An LLM is not. Tiny changes in prompt, model version, or context produce different outcomes. Vendors swap model weights. Your knowledge base drifts. Here are some examples of how things can go wrong with LLMs in spectacular ways!

“The Friday Rollout That Looked Fine in Staging” You upgrade a prompt and bump the temperature from 0.2 → 0.4 to make your assistant feel friendlier. Unit tests still pass. By Monday, support volume is up 40% because:

• The assistant started adding confident-but-wrong “extra context.”

• Token counts rose ~25% due to chattier answers.

• Latency crossed your SLO during peak hours.

Here’s another classic:

“Temporaly Confused Bot”

Your chatbot answers tax questions. A user asks: “What’s the current VAT rate in the UK?” The model reads an old PDF from 2019 and replies with a past rate. The user posts the wrong answer on social. Support wakes you up. Fun times.

But how to avoid such “user report → war room → roll back” drama?

Evals: measure quality in plain numbers

Evals turn “feels right” into metrics you can track. Do both offline (pre-deployment) and online (shadow tests or sampled live traffic).

Factual Accuracy Test if answers match your provided docs. Build small Q&A sets per collection and score with a simple rubric: “fully supported,” “partially,” “not supported.” Log which passages the model cites.

Temporal Accuracy Test if the model gives the current truth. Tag questions as time-sensitive. Keep a small table of ground truth with dates (e.g., “VAT = 20% as of 2025-01-01”). If the answer is old or hedges, it fails.

Here are some other evals that can be useful:

• Format Validity: Is the JSON valid? Does it match your schema?

• Safety & PII: Refuses risky requests; redacts emails/IDs when required.

• RAG Faithfulness: Is the answer supported by retrieved text? Penalize made-up facts.

Tracing:

• Make every run debuggable (Logging)

When something goes odd, you need a clear trail from input to output. This is on top of a classic logging system since LLMs have new requirements like what model config was used.

Here is a suggested list of what to record on every request:

• Prompt version and all variables filled in

• Model name, temperature, max tokens

• Retrieval query, top-K, and the actual chunks shown to the model

• Every tool call: name, args, result, errors

• Guardrail checks and why they passed/failed

• Token counts (prompt, completion)

• Latency per step and end-to-end

• Caching, retries, and fallbacks taken

• No surprises on the bill (Cost Monitoring)

LLM cost is mostly tokens and tool calls. It can spike fast.

Track spend by model, endpoint, feature, team, and user. Watch:

• Tokens per request, and per successful task

• Cost per 1K requests and per solved ticket

• Tool call density (some chains spam tools)

• Cache hit rate (misses are expensive)

Add budget alerts and soft limits. Route simple tasks to a smaller model. Use a “tiny prefilter → big model on hard cases” path. Compress context, prune long histories, and store reusable summaries.

• Fast models feel smarter (Latency)

Speed is part of quality. Users judge the answer and the wait.

Measure latency at each hop:

• Retrieval time

• Model time

• Tool time (each call)

• End-to-end time, with p50/p95/p99

Tune by caching hot results, prefetching facts, lowering top-K, streaming tokens to the UI, and moving slow tools off the critical path. Set alerts on p95/p99, not just averages.

Tooling Options:

You can build from scratch, but most teams mix in ready tools for LLMOps. Here are some popular picks:

Tip: Most teams start with Langfuse (or Helicone) for traces + basic evals, add Humanloop for richer human-in-the-loop workflows, and integrate LangSmith if they’re already heavy on LangChain.

Wrap-up

LLMs are powerful, but they wander. LLMOps Monitoring keeps them in bounds. With evals, tracing, and clear cost/latency views, you find issues before your users do. Start small, measure what matters, and keep your model on a short, friendly leash.

It's now widely recognized that neglecting to design the blank slate is a big mistake because it's the user's first impression of your product. You might have the best user interface when the app is filled with data, but users may never reach that stage if it's not clear how to interact with the application when it's empty.

This is a fascinating concept, and as I was reflecting on how organizations evolve, I thought that many business leaders could take a page from the designer’s book about the importance of “blank slates” in their plans. Sure, when you step into a new role, tasked with transforming a department, it’s crucial to have a vision for how teams will interact in the future, which roles to introduce, which profiles to hire, and who will be responsible for what. However, a leader should not bypass the design of the “blank slate”, as in, the snapshot of what the organization looks like at the beginning of this transformation.

Too often, leaders can’t resist the temptation of dictating their vision from an ivory tower, while hiring a layer of smart middle-managers and delegating the execution of the plan to them. But even the most brilliant hires will get lost in the mess if they are shown a picture that only exists in the head of their enlightened boss. Imagine welcoming a new engineer into a maze of legacy services nobody remembers how to sunset. She receives a visionary slide describing a “data mesh” utopia, yet her onboarding checklist is full of monoliths that refuse to die.

In the gulf between present reality and promised future, talent grows frustrated, momentum stalls, and politics fill the gaps. Legacy services keep getting more and more consumers without a clear horizon for deprecation. Teams and people who struggle to see themselves in the future picture fight to stay relevant, trying to expand their area of influence until someone with a mandate tells them to stop.

Designing that organizational blank slate means scripting the first moves, not just day‑three‑hundred outcomes. Leaders should map out temporary but explicit ownership zones, identify what must be kept running at all costs, and label the placeholders that are expected to disappear. Much like the “Add your first project” card in an empty dashboard, these markers tell every newcomer, “Here’s where to start making an impact, and here’s when this chunk of work should gracefully sunset.”

In the data engineering world, bigger usually means better. More power, more scalability, and more buzzwords! PySpark, for instance, is typically our trusty steed when tackling massive datasets on distributed clusters like Databricks. But what if, in our race to scale everything to infinity, we've overlooked simpler, faster solutions?

Here's a story where simplicity won big—and PySpark learned a lesson in humility.

The Great Excel Report Battle

I was tasked with creating an Excel report based on a moderately sized dataset—around 100,000 rows. My initial weapon of choice? PySpark DataFrames running in Databricks. After all, PySpark thrives on handling large datasets, and our environment was already set up. Easy choice, right?

Well, not exactly.

The report involved filtering data by certificate types, applying various transformations and sorting criteria, and finally writing each filtered set to separate Excel sheets (132 sheets in total). Here's a glimpse of the PySpark code I confidently started with:

def write_dataframe_to_excel(writer, df, certificate):
    df = df.withColumn(
        "Status Priority",
        F.when(F.col("Status") == "Non-Comply", 0)
        .when(F.col("Status") == "Warning", 1)
        .when(F.col("Status") == "In-Future", 2)
        .otherwise(3)
    ).withColumn(
        "Sorted Expiry Date",
        F.coalesce(
            F.to_date(F.col("Certificate Expiry Date"), "dd/MM/yyyy"),
            F.to_date(F.lit("01/01/1900"), "dd/MM/yyyy")
        )
    ).orderBy("Status Priority", "Sorted Expiry Date")
    df = df.drop("certificate_code", "certificate_name", "Compliance Status", "Status Priority", "Sorted Expiry Date")
    df.toPandas().to_excel(writer, index=False, sheet_name=certificate, startrow=2)

with pd.ExcelWriter(excel_path, engine="openpyxl") as writer:
    for certificate in main_certificate_codes:
        df = compliance_df.filter(F.col("certificate_code") == certificate)
        if certificate in ["BOSIET", "HUET"]:
            df = add_caebs_columns(df)
        df = df.filter(F.col("Compliance Status").isin(["Non-Comply", "Warning", "In-Future"]))
        write_dataframe_to_excel(writer, df, certificate)

Initially, it looked perfect. But execution took nearly 30 minutes! How was PySpark, the champion of big data, struggling with a modest Excel report?

Enter Pandas: The David to PySpark’s Goliath

I started suspecting that I might be over-engineering my solution. After all, was distributed computing even necessary for a dataset of just 100,000 rows?

So, I decided to rewrite the logic using Pandas DataFrames, bypassing PySpark’s overhead entirely. The results were nothing short of astonishing:

def write_pd_dataframe_to_excel(writer, df, certificate):
    df["Status Priority"] = df["Status"].map({
        "Non-Comply": 0,
        "Warning": 1,
        "In-Future": 2
    }).fillna(3)
    df["Sorted Expiry Date"] = pd.to_datetime(
        df["Certificate Expiry Date"], 
        format="%d/%m/%Y", 
        errors="coerce"
    ).fillna(pd.to_datetime("1900-01-01"))
    df = df.sort_values(by=["Status Priority", "Sorted Expiry Date"])
    df.drop(columns=[
        "certificate_code", "certificate_name", "Compliance Status", "Status Priority", "Sorted Expiry Date"
    ], inplace=True)
    df.to_excel(writer, index=False, sheet_name=certificate, startrow=2)

compliance_df = add_caebs_columns(compliance_df)
compliance_df = compliance_df.filter(
    F.col("Compliance Status").isin(["Non-Comply", "Warning", "In-Future"])
)
pd_df = compliance_df.toPandas()
pd_df["PIN"] = pd.to_numeric(pd_df["PIN"], errors="coerce")
with pd.ExcelWriter(excel_path, engine="openpyxl") as writer:
    for certificate in main_certificate_codes:
        df = pd_df[pd_df["certificate_code"] == certificate]
        if certificate not in ["BOSIET", "HUET"]:
            df = df.drop(columns=["CAEBS Issue Date", "CAEBS Expiry Date"])
        write_pd_dataframe_to_excel(writer, df, certificate)

This Pandas solution reduced execution time from 30 minutes to just 2 minutes! No distributed cluster, no overhead, just straightforward Python and Pandas magic.

Why Pandas Crushed PySpark in This Case

Why did Pandas outperform PySpark by such a massive margin? The key lies in the nature of the data and the overhead associated with distributed computing.

• Overhead of Distributed Processing: PySpark is designed for distributed computing across multiple nodes, involving tasks like data serialization/deserialization, task distribution, and network communication. For smaller datasets like ours, this overhead becomes a significant burden, drastically affecting performance.

• Absence of JVM Overhead: PySpark runs on the Java Virtual Machine (JVM), which introduces additional overhead and complexity. Pandas, being purely Python-based, avoids this altogether.

• In-Memory Operations: Pandas operates entirely within memory on a single node, making computations significantly faster.

• Data Locality: Pandas capitalizes on data locality, ensuring minimal data movement and latency, enhancing processing speed significantly compared to PySpark's distributed nature.

Has Pandas ‌Been Abused In Data Engineering?

Well No! However, Pandas won the game in this scenario by margin, it isn't always the optimal solution. Pandas have some breaking points that might make to to think of other game players like Polar or PySpark:

• Memory Constraints: Pandas loads all data into RAM. If your dataset size exceeds available memory, Pandas can break or severely slow down performance.
• Complex Joins on Large Data: Performing complex joins or operations on large datasets might exceed Pandas' capabilities, causing performance degradation.
• Parallel and Distributed Processing: Pandas is single-threaded, and If parallel and distributed computation is necessary (like ETL on massive datasets), Pandas doesn't scale well.
• Large-scale I/O Operations: Pandas struggles with massive read/write operations, particularly from distributed sources or formats like Parquet, Avro, or ORC.
• Real-time Data Processing: Pandas is not optimized for streaming or real-time processing, where PySpark excels.
• Columnar Optimization: Unlike Polars or PySpark, Pandas doesn't have built-in support for optimized columnar operations, limiting speed in certain operations. If you have lots of columnar operations on a large dataset, Pandas is not a good choice.

Lessons Learned: Choose Your Tools Wisely

This isn’t a "Pandas always beats PySpark on small datasets" scenario—it’s about picking the right tool for the job. Before going for a tool, you need to consider these factors holistically:

• Data Volume: Does your dataset comfortably fit in RAM?
• Computation Type: Are you performing complex transformations or simple operations?
• Latency Requirements: Is real-time or near-real-time performance critical?
• Infrastructure Available: Do you have access to distributed computing resources?
• Complexity of Joins and Aggregations: Are complex operations frequent?

Use Pandas for quick, ad-hoc tasks on small-to-medium datasets. Opt for Polars for medium-large datasets on a single powerful machine. Choose PySpark for distributed computing, large datasets, or real-time analytics.

Below is a quick comparison table:

Simply storing data isn’t enough. To truly unlock its value, organizations need to treat data as a product: something that is deliberately designed and made usable for specific business needs.

At first glance, this may sound like additional overhead. Another layer of governance? That’s going to slow our initiatives down! But in reality, the opposite is true. Consider the amount of time spent questioning data sets when they are dumped in a centralized system. How many times have you asked yourself questions like: Is it really the data set we need? Oh wait, this column actually means something else in our data model. The data in this table seems incomplete. I can’t understand what it represents. The schema changed again, now our system is broken in production…

Modern frameworks have proposed alternative ways of managing data sets. But it all stems from a different way of thinking about your data. In this post, we’ll break down the key concepts behind data as a product, the difference between data sets and data products, and how this shift improves usability, accessibility, and governance.

Understanding the Basics: Data, Data Sets, and Products

Before diving into data as a product, let’s clarify some fundamental terms:

• Data: The raw facts and figures collected by an organization—numbers, text, timestamps, sensor readings, etc.

• Data Set: A structured collection of related data points, typically stored in formats like tables, CSV files, or databases. Examples include a table of customer transactions, machine sensor logs, or financial records.

• Product: Something deliberately designed and built to provide value to a user, whether physical (a phone) or digital (a mobile app).

• Data Product: A combination of data set(s), domain model, and user experience, designed for a specific use case. Examples include a fraud detection model based on transaction data, a self-service analytics dashboard for sales teams, or an API that provides real-time customer insights.

Data Set vs. Data Product: What’s the Difference?

Let’s start with an example and consider the sales data in an e-commerce business.

• A data set might contain raw customer transactions with thousands of line items. Without context, a user might struggle to extract insights. The transactions don’t serve any purpose by themselves: they are simply a collection of information.

• A data product could be a "Monthly Sales Performance Dashboard", which combines one or more data sets (the raw transactions), the domain model (aggregation into revenue trends, regional breakdowns, …) and the user experience (the graphic UI of the dashboard, its access policies, the ability to drill down in a report, …).

A data set is a fundamental component of a data product. But a data product is much more than that. It must include:

✅ Defined ownership – Who is responsible for maintaining the data?

✅ Clear documentation – What does the data mean? How should it be used?

✅ Usability – Is the data structured in a way that users can access and understand?

✅ Ongoing updates & maintenance – Is the data kept fresh and accurate?

Business Domains Define Data Products

Organizations don’t need just data. They need data that serves a purpose. That’s why defining data products should always start from the business domain:

🔹 Finance teams may need a revenue forecasting data product based on historical transactions.

🔹 Marketing teams may need a customer segmentation data product that enriches demographic data with purchase behavior.

🔹 Operations teams may need a real-time logistics dashboard that tracks shipment statuses and delays.

By starting from business domain needs, companies can design data products that are immediately useful, rather than dumping raw data into a central repository and expecting teams to figure it out on their own.

What Does It Mean to Treat Data as a Product?

A data product must be designed, maintained, improved, and eventually retired when no longer needed. This approach ensures that data remains valuable and does not become obsolete. The whole lifecycle should revolve around the following key principles that augment the world of “data sets” by introducing product thinking as a new dimension.

Usability & Customer-Centricity

🔹 Who needs this data? How will they use it?

A well-designed data product is intuitive and built for the end user. This means:

✅ Well-documented definitions so users understand the data.

✅ Consistent structure and formatting to avoid confusion.

✅ Version-controlled updates to prevent disruptions in workflows.

Unlike traditional data management, which prioritizes storage and availability, a product mindset ensures that data is packaged for real-world usage, just like a consumer-facing app or tool.

Accessibility

🔹 How easily can users find and interact with the data?

Many organizations struggle with data locked away in silos, making it difficult for teams to access and use. A data product should be discoverable and well-integrated, for example it should have:

✅ Self-service data catalogs to remove unnecessary friction between teams and questions such as “Could you tell me which data sets are available?”

✅ Role-based permissions to manage security and compliance.

✅ Standardized formats to allow smooth integration with other tools and platforms.

By ensuring discoverability and controlled access, data becomes a trusted, reusable resource instead of a hidden asset that requires manual extraction.

Pragmatic Governance with Data as a Product

One of the biggest challenges with traditional data management is poor governance—unclear ownership, inconsistencies, and compliance risks. Treating data as a product solves many governance issues by design.

1. Clear Ownership & Accountability

• Every data product has a designated owner (often a business or data team) responsible for its accuracy and updates.

• Unlike traditional IT-driven data models, ownership is distributed across business domains. This ensures, for example, that data issues are addressed by people who have a deep understanding of the business processes that generate the data itself.

2. Built-in Data Quality

• Errors and inconsistencies are addressed before reaching users, thanks to holistic observability.

• Data contracts ensure that data sources follow a standard schema and remain compatible across different systems.

3. Stronger Security

• Access controls are built into data products, preventing unauthorized access while facilitating access for authorized users.

• Automated audit logs track data usage, making compliance with GDPR, CCPA, and other regulations easier.

📌 According to a 2024 Gartner survey (Evolution of Data Management), the top investment trends for the next three years are AI-ready Data Initiatives and Data quality & Governance. Treating data as a product supports both: AI models require structured, well-maintained data, and governance naturally improves when ownership and usability are prioritized.

Data as a Product is a Transformation, Not Just a Definition

Shifting to data as a product is not just a matter of changing terminology—it requires a fundamental transformation in how an organization creates, manages, and uses data. This shift impacts business processes, technology, governance, and culture.

The transition can be challenging, especially in organizations with legacy systems and siloed data ownership. More importantly, the biggest hurdle is a culture that treats data as an IT responsibility rather than a business enabler.

If you're looking to adopt a data-as-a-product approach and need guidance on where to start, get in touch. We can help you design a strategy that fits your business and unlocks the full value of your data. 🚀

Here’s a story on the transformative journey of Data as a Product at one of our past clients.

Large Language Models (LLMs) have become ubiquitous, but their tendency to “hallucinate” non-factual details remains a challenge. Retrieval Augmented Generation (RAG) was introduced to ground LLM responses in actual data by retrieving relevant facts and incorporating them into prompts. However, when your data is highly interconnected—such as in codebases or scientific documents which call and cite other parts of the data respectively —traditional vector-based search can miss key relationships.

In this post, we’ll explore how integrating Knowledge Graphs into the RAG pipeline can boost the relevance and factual accuracy of the output. We’ll compare a conventional VectorDB RAG (using Amazon Bedrock with OpenSearch) with a Graph-Powered RAG (using Amazon Bedrock alongside a graph database like Amazon Neptune or Neo4j).

Summary YouTube Video:

• Will be Published Soon!

Problem Statement

While RAG helps mitigate LLM hallucinations by providing factual context, its typical implementation using vector similarity search has a critical flaw: It looks for similar chunks of document to add to the context and “similar” does not always mean “relevant”! For instance, when asking, “What are the possible causes of obesity?” a vector search might return descriptive text about obesity rather than pinpointing factors like “hypothyroidism” or “bad diet”—the actual causes.

This issue was tried to solve using various techniques, most notably ReRanking of the RAG result using another LLM based on relevancy. But one solution we are interested in is using Knowledge Graphs. We wrote a series of blog posts on Knowledge Graphs: Our Blog Series. Here we want to see if adding them to the LLM pipeline can help the performance of the generation.

Fig 1. Knowledge Graph-Powered RAG vs Vector RAG

Use Case Example: Code Refactoring

Consider a codebase, here the Python Requests Library, where functions are scattered across multiple files and are highly interconnected. Suppose you need to refactor a function (say, utils.urldefragauth), but you’re unsure which other functions depend on it. A simple vector search might return the function definition only. However, a Knowledge Graph that maps function dependencies can retrieve all related functions—giving you the necessary context to ensure a safe refactor.

Knowledge Graph-Powered RAG: How Does It Work?

Traditional RAG uses vector embeddings to find similar text chunks. In contrast, Knowledge Graph-Powered RAG leverages a Knowledge Graph to capture explicit relationships among data points (e.g., function calls in code). This approach uses a query language—such as Cypher—to ask precise questions about how entities are connected.

For example, a valid Cypher query to find which functions call a particular function might be:

MATCH (caller:Function)-[:CALLS]->(target:Function {name: 'utils.urldefragauth'})

RETURN caller.name AS CallingFunction, caller.code AS CallingFunctionCode

LIMIT 5;

Here I used the LlamaIndex package which provides some functionality including the TextToCypherRetriever which does the heavy job of turning a text query into a Cypher query compatible with your knowledge graph. What happens underneath it though is basically begging an LLM to turn the query and knowledge graph schema into a valid Cypher query!

How to Prepare the Code Graph?

Code repositories are a perfect example of highly interconnected data. Modules import functions from various other modules and are themselves called by various other functions.

A crucial step in our workflow is preparing a fact-based Knowledge Graph—in this case, a graph of functions and their relationships, specifically which function calls which. Here, I used the Python ast library to parse the Python files of the popular package requests.

Here are some code snippets:

def parse_python_file(file_path):

    """Parse a Python file to extract functions, classes, dependencies, imports, and code."""

    with open(file_path, "r") as f:

        source = f.read()

        source_lines = source.split('\n')

        tree = ast.parse(source, filename=file_path)


    visitor = FunctionVisitor(source_lines)

    visitor.visit(tree)

    return visitor.functions, visitor.imports, visitor.classes

Below is a part of the final graph (stored in Neo4j AuraDB):

Graph-Powered RAG vs Vector Database RAG: Example

To show the difference between a vector database approach and a knowledge graph approach, I set up an AWS Bedrock Knowledge Base, which uses OpenSearch Service for vector search. So I will compare its results with the Knowledge Graph hosted on a Neo4j database (alternatively, I could use an Amazon Neptune database for Graph with some small changes in the code!).

I asked both of these workflows (or you can call them AI Agents!) what happens if I change a function? This question clearly shows the difference in the graph approach, since to answer, you need to know which other functions call this function, which can be in any other files!

Here is the input to both of these agents:

I want to refactor the utils.urldefragauth function. What other functions call it and might get affected? Show the CODE to those function.

Graph-Powered RAG Answer (Generate using Amazon Bedrock):

Based on the provided information, the function that calls utils.urldefragauth is adapters.HTTPAdapter.request_url. This function might be affected if you refactor utils.urldefragauth. Here's the code for adapters.HTTPAdapter.request_url:

def request_url(self, request, proxies):

    """Obtain the url to use when making the final request.
    If the message is being sent through a HTTP proxy, the full URL has to
    be used. Otherwise, we should only use the path portion of the URL.
    This should not be called from user code, and is only exposed for use
    when subclassing the
    :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
    :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
    :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs.
    :rtype: str
    """
    proxy = select_proxy(request.url, proxies)
    scheme = urlparse(request.url).scheme
    is_proxied_http_request = proxy and scheme != "https"
    using_socks_proxy = False

    if proxy:
        proxy_scheme = urlparse(proxy).scheme.lower()
        using_socks_proxy = proxy_scheme.startswith("socks")

    url = request.path_url
    if url.startswith("//"):  # Don't confuse urllib3
        url = f"/{url.lstrip('/')}"

    if is_proxied_http_request and not using_socks_proxy:
        url = urldefragauth(request.url)

    return url

This function uses urldefragauth when dealing with proxied HTTP requests that are not using SOCKS proxy. Any changes to the behavior or signature of utils.urldefragauth could potentially impact this function, so you'll need to ensure that your refactoring doesn't break the existing functionality or interface expected by adapters.HTTPAdapter.request_url .

Vector Database RAG Answer (Generate using Amazon Bedrock):

I apologize, but I cannot provide an answer to your question based on the given context information. The provided code snippet does not contain any reference to a utils.urldefragauth function or any functions that call it. The code appears to be a collection of utility functions related to networking, proxies, and HTTP requests, but it doesn't include the specific function you're asking about or any functions that call it. Without more relevant context about the utils.urldefragauth function, I'm unable to identify the functions that might be affected by refactoring it or show you their code.

Comparison:

As you can see since the Vector database is unable to see the called functions relationship it was unable to answer the question unlike the Knowledge Graph approach which successfully found the relationship and answered the questions!

Table Summary:

Conclusion

For applications where factual accuracy and interconnected data are critical, a Graph-Powered RAG approach shows considerable promise. Although it demands a higher upfront investment in terms of graph preparation and system complexity, the improved contextual accuracy can be invaluable—especially for domains like code refactoring or multi-entity document retrieval.

For those interested in experimenting further, the full code repository is available here:
GitHub Repository: CodeGraphRAG