Nobody Could Tell Us Why the Number Was Wrong

The ticket that takes three days

Monday morning, the data support queue looks like this:

• Finance: "Revenue on the dashboard is $7.2M. Our month-end sheet says $8.1M. Which is right?"
• Sales: "My region's numbers dropped overnight. Did we really lose deals?"
• Operations: "Yesterday's shipments still show zero. Is the report broken?"
• A store manager: "This total doesn't match what I see in the ordering system."

None of these people are technical, and they shouldn't need to be. They just want to know whether they can use the number in front of them.

Each ticket lands with a data engineer. The engineer drops what they're doing and spends hours, sometimes days, tracing that one number back through five different systems. The answer is usually boring: a hidden filter, a job that failed overnight, data that hadn't refreshed yet, or two teams using two meanings of "revenue".

Then the same kind of ticket comes in again next week, from someone else, and the whole investigation starts from zero. Meanwhile people quietly stop trusting the dashboards and go back to their own spreadsheets.

I have been a data engineer for over nine years, and I have worked a lot of these tickets. So I set out to build a support engineer that never sleeps: one that picks up the ticket, finds out why the number is what it is, answers in plain words, and remembers the pattern so the next one is instant.

This is the honest story so far: what I tried, what failed, what I got wrong, and what I'm doing next.

Why nobody can answer

A dashboard looks simple: a box with a number in it. But that number has made a long journey before it reaches the screen.

In a typical company it goes roughly like this:

1. It starts in a business app, such as the billing or ordering system.
2. It is copied into a raw storage area every night.
3. It is cleaned: duplicates removed, currencies converted, cancelled items dropped.
4. It is shaped for business use: totals, months, customer groups.
5. A reporting model adds formulas: "revenue means only active subscriptions".
6. The dashboard shows it, often with filters some viewers never see.

Every step is built by a different person, at a different time, with its own rules. Each step runs on a schedule, and any of them can fail or run late.

So when two numbers disagree, the answer could be hiding in any of six places. Finding it means opening each one, reading code, and re-counting the data by hand. That's why it takes days.

The tools companies buy are very good at showing the number. None of them can explain it.

What I set out to build

The idea is simple to say: an AI investigator that does what that engineer does for two days, but in minutes, and shows its work.

You tell it which number looks wrong and what you compared it with. It then:

• Rebuilds your number, so it sees exactly what you saw.
• Rebuilds the other number, the one from the spreadsheet or the other report.
• Walks back through each step of the journey and finds where the two numbers split apart.
• Finds the reason: a filter, a failed job, a late refresh, a different definition.
• Answers in plain words, with the evidence attached so an engineer can check it.

From the start I set three rules:

1. It can only look, never touch. It has read-only access, enforced in code, not just by asking the AI nicely.
2. No guessing. Every claim must come with the query and the numbers that prove it.
3. Costs stay capped, so it can never run up a surprise bill.

I built it with AI coding agents doing most of the programming, working in rounds, while I designed, reviewed and steered. I tested it on practice company setups that I built myself, so no real company data was ever involved.

"Doesn't Copilot already do this?"

It's a fair question, and the honest answer is: partly, for some people, inside one platform.

The AI assistants built into data platforms are improving fast. Fabric's data agents can answer questions across up to five of a company's reports and data stores inside Fabric. Copilot can explain a failed pipeline's error message when an engineer opens it. Databricks announced Genie ZeroOps in June 2026, an agent that traces pipeline failures to their root cause; it is heading into private preview.

They are good tools, and they are getting close to this problem. In a Databricks summit session, an engineer fixes a broken revenue dashboard in about two minutes, inside Databricks. What they don't cover yet:

• Most numbers cross platforms. A real number often travels from a business app, through one or two data platforms, into a reporting tool from another vendor. Each vendor's assistant mostly sees its own part of that journey.
• They need a seat for every user. A company may have thousands of people who read dashboards. Giving all of them an AI license and data access, just to ask the occasional "is this right?", is expensive and risky. A support desk needs no seat.
• They don't remember the pattern. The same question from ten people is ten separate conversations, not one known issue.
• They start from the engineer's side: a failed job, a broken table. A support ticket starts from a person who only knows that two numbers don't match.

So I see this as filling the gap between them, not competing. It works from the support desk: a ticket comes in, from anyone, and gets an answer. For companies that already use an AI assistant, whether Copilot, Claude or ChatGPT, it can plug in behind it, so asking "why is this number wrong?" there gets a real answer too.

What worked

The foundations came out well. On my main demo ticket, about warehouse units handled, the tool traced a disputed number across two layers (7,661 units at one step, 8,765 at the next) and pointed at the step in between: a table join that can count some rows more than once. I graded that "partly answered": right place, likely cause, not yet proven with the actual duplicate rows.

Around that:

• The safety rules hold. Every query goes through a gatekeeper that blocks anything except reading. Tests actively try to sneak write commands past it, and they are all blocked.
• A map of the whole system. The tool builds a map of how every number on every report connects back to its source: about 1,750 connections for one practice company. It keeps that map current by checking for changes every six hours.
• It works on a setup I didn't build, partly. I pointed it at a sample company from Microsoft, to make sure it wasn't secretly tuned to my own examples. It mapped 396 connections. It understood 192 of them; the other 204 it marked as unknown, each with the reason. Most often (119 of the 204) it was a report column it couldn't trace to the storage table underneath.
• The plumbing is well tested. There are over 4,400 automated tests, mostly written by the AI agents. They check the safety rules, storage and replay. They do not measure whether the answers are right.

So the foundations are solid, but the tool is not yet answering real questions reliably. The next section is why.

What failed

It couldn't read the question. This is the biggest failure. The only time I scored the tool on fresh tickets it hadn't seen, it got 0 of 12. None of them even reached the investigation. Six crashed while the tool was turning the ticket into a structured question. The other six got stuck asking the person clarifying questions, some of which a human support engineer would never ask.

One caveat, stated honestly: the 12 tickets were written by an AI tester, all about the same report card. I haven't yet checked by hand whether every one was fair and answerable. Even so, crashing on half of them is a real failure, not a testing artifact.

The cloud demo failed again and again. A real product can't live on one person's laptop, so I moved it to the cloud. Over three rounds of work it has not passed once. Each attempt broke on something different: a budget counter, a missing setting, a timeout, a login error on one data layer, a file path, a badly written header. One time the check actually ran, and the result was lost because the step that saves it crashed.

Not one of those failures came from the investigation logic. They were all plumbing.

The platform kept its secrets locked. To explain a number, the tool needs to read how each report and formula is defined. In practice, the platform only shows those definitions to accounts that are allowed to edit them. My tool is deliberately read-only. So I had to create a separate, tightly fenced scanning account just to read the blueprints, with code that blocks it from doing anything else.

"Prove it" became too hard to prove. I asked the tool to prove each connection on the map with real data. It refused to call anything proven unless it could replay every filter through every layer exactly. That's a good long-term goal, but it meant zero paths were marked as proven, even where the numbers clearly matched.

The recordings ate my disk. To make every run repeatable, the tool recorded everything it did. Those recordings grew past 200 GB on a laptop. I spent real time moving them to cheap cloud storage and cleaning up instead of building.

Where I went wrong

I made it strict before I made it work. I wanted a trustworthy tool, so I added checks everywhere: six different budget counters, a limit of one or two attempts per test, and rules that every run must replay perfectly before the next could start. Each check made sense on its own. Together, they meant any tiny mismatch stopped everything, and fixes no longer fit. In a test environment you need room to fail, retry and fix. Strictness belongs at the end, once things work.

I kept starting runs by hand. Each cloud attempt was launched with a slightly different one-off script. Every new script carried a new small mistake. A product needs one boring, tested way to start a run, used every time.

I underestimated the boring layer. I compared my tool to how ChatGPT reads someone's bank transactions through Plaid and sorts them neatly. That feels like magic, but the magic is mostly Plaid: years of work turning every bank's messy data into one clean format. The AI only reads clean data. My tool had no Plaid underneath. It had to collect and clean the facts itself, every time, while also answering the question.

I asked the tool to understand too much, too exactly, before doing anything. The intake step tried to pin down every detail of a ticket (which report, which page, which filters, which direction the numbers should move) before the investigation could start. If anything was missing or contradictory, it either crashed or asked another question. Real people write short, vague tickets. A good support engineer starts with what's there, checks the obvious things first, and asks one clear question only when they're truly stuck.

The fix has three parts. Make the few essentials (which report, which number, compared with what) required when the ticket is filed, so the tool doesn't have to dig for them. Allow at most one clarifying question, with choices to pick from. And, most importantly, check the facts before trying to understand the question in depth, which is the plan below.

The lesson: the hard part of an AI product is rarely the AI. It's reliable access to clean facts, and the discipline to keep the plumbing simple.

What I'm doing now

The new plan is to build my own "Plaid" first: a layer that quietly collects facts, so the AI mostly reads instead of digs.

First, make it boring. Loose rules while building, with hard stops only for safety: any attempt to change data, a leaked password, or the monthly budget running out. Everything else gets recorded, fixed and retried. One tested command starts every run, on a laptop or in the cloud. A small health check runs every morning.

Then, collectors and a facts ledger. Small scheduled programs wake up every 15 to 60 minutes, ask "what ran, what failed, what refreshed, what changed?", write the answers down with a timestamp, and go back to sleep. No AI involved. Over time this builds a timeline of facts about the whole system.

Then, one investigator that reads the ledger first. Most questions turn out to be boring, so most answers become instant:

"Why is yesterday's revenue low?" → "The overnight load failed at 2:40 am. The next run is at 6:00."

This is also the answer to the 0 of 12. In my experience, most real tickets come down to a late or failed job, and for those the tool barely needs to understand the question: the ledger already knows what went wrong on that report's path. Deep understanding is only needed for the genuinely odd cases, and those get the full walk back through the layers.

Collectors fill the ledger in the background; a question only goes deep when the facts can't explain it.

Every answer ends in one plain verdict:

• As designed: a rule in the logic explains the difference, and here is the rule.
• Not caught up yet: the source is ahead; the next scheduled run will carry it.
• Waiting on a job: a scheduled run is late or queued.
• A job failed: which one, when, and the error.
• Report not refreshed: the data is current, but the report isn't.
• Different question: the two numbers use different dates, filters or scope.
• Different access: people with different permissions see different rows.
• Source changed history: someone corrected or back-dated records in the business app.
• Not found: it says exactly where the numbers split, and admits it couldn't find why.

After that come the parts that make it a real support engineer: picking up tickets where people already file them, and spotting repeated issues so they get fixed once. I'll write about those when they're built, not before.

Why I'm sharing this now

Dashboards tell you what the number is. Nobody tells you why it's that number, or whether you can trust it. That gap costs companies days of engineering time and, worse, quietly erodes trust in data altogether.

I'm sharing the failures as well as the progress, because most of what I learned came from them. If you build AI tools: the model is rarely the bottleneck. Clean facts underneath and simple plumbing around it matter more. And if you work with data: you have probably sat in that meeting too.

Next up is getting the cloud version to pass end to end, then building the facts ledger. I'll write about what happens, whether it works or not.

If you've lived through the "which number is right?" meeting, I'd love to hear how your team handles it.