Work · Open source · 02
faultline
When a dbt pipeline breaks, what broke it, and where is the evidence?
In 30 seconds
- Problem
- When a dbt pipeline breaks, an engineer digs through tests, lineage and commits for hours to find what broke it.
- Result
- Named the exact root cause in 43 of 48 dev incidents (89.6%), with 0 confidently wrong answers and 0 false alarms on 30 clean runs. Held-out score: pending.
- My role
- Sole developer: fault injector, 192-incident benchmark, eval harness, monitors and triage.
- measured 89.6% root cause found at top-1 on dev (43 of 48 incidents)
- measured 0 of 48 confidently wrong answers on dev; 0 of 30 false alarms on clean runs
- measured 192 injected incidents across 8 fault types, plus 70 clean runs
- pending — held-out score: the frozen test split has not been run yet
- Type
- Open source
- When
- November 2026 · in progress
- Role
- Sole developer: injector, benchmark, eval harness, monitors, triage
- Scale
- 4 dbt projects · 193 tests · 5 methods scored
- Status
- Milestone 5 of 8 · benchmark v1 released
01In plain language
Data teams build their reporting tables with dbt. When a number on a dashboard is suddenly wrong, someone has to find out why: bad data arriving, a column renamed upstream, a join that started duplicating rows, or a code change last Tuesday.
faultline does that search. It watches the tables, groups the symptoms into one incident, proposes possible causes, and tests each one against the recorded data. It only names a cause that passed its test. If none did, it says "not confirmed".
To know whether that works, I first built the exam: a tool that breaks real dbt projects on purpose, in eight different ways, and records the right answer for every break.
02The question
dbt's own tests catch some failures. How many root causes can a system name exactly, down to the table, column and commit, without ever pointing confidently at the wrong one?
03Why the obvious approach fails
dbt tests are the standard safety net. On the dev split they raised an alert for 22.9% of incidents and named the cause for none. A test tells you that a table is wrong, not why.
Pasting the logs into a language model is the popular shortcut. With a local 7B model, it named the right cause once in 48 incidents and a wrong cause seven times.
Both share the same gap: neither checks a guess against the data before saying it.
04The gate
Every decision goes through the same order: deterministic lineage and checks first, then statistics, and a language model last. The model's only planned job is to rank causes that have already passed a test and to write the report. No raw rows ever reach a model.
| Layer | Job | State |
|---|---|---|
| Lineage | Table and column lineage from compiled SQL; what feeds what | Built |
| Monitors | Seven checks: schema, volume, freshness, null rate, distribution, duplicates, time shift | Built |
| Triage | Group symptoms, propose causes, test each one, keep only survivors | Built (deterministic) |
| Local LLM | Re-rank survivors and write the report | Next milestone |
The headline number on this page comes from the deterministic layers alone. The model has not been added yet, so it cannot have helped.
05Thought process
Left to right, the path one incident takes. Each card hands the next one less uncertainty.
- 01
Gather
- dbt artefacts and run logs
- Monitor anomalies, failed tests
- Lineage of this run, cached
Symptoms, not guesses.
- 02
Group
- Symptoms joined by table lineage
- One incident, not ten alerts
- Stored in DuckDB
One problem, one report.
- 03
Propose
- Data causes at seeds and sources
- Code causes at changed models
- With the commit that changed them
Every place it could have started.
- 04
Test
- One check per cause
- On aggregates, lineage, SQL, history
- Failing causes are dropped
Guesses must survive the data.
- 05
Rank and report
- Survivors only
- "Not confirmed" if none survive
- LLM re-ranker: next milestone
Never a confident wrong answer.
06Decisions, and what they cost
A fault injector and 192 scored incidents came before any diagnosis code.
Hand-picked examples always work. A benchmark tells you how often.
The jira split (58 runs) was generated by me on my own machine and never opened in the coding session. A CI guard blocks any code that names it.
Whoever writes the rules must never see the final exam.
A cause counts only if the table, the column (when there is one) and, for code faults, the commit are right.
The table-only score is reported, but only as a diagnostic, never as the headline.
A code mutation is kept only if the build fails or a table profile changes.
A change with no symptom has no correct answer and would only add noise.
When compiled SQL reads a table dbt did not declare, the edge is kept and reported.
The SQL is what runs. The Stripe project has two such hidden reads.
07What came out
| Fault type | n | Top-1 | Wrong top-1 |
|---------------------|----|----------------|-------------|
| schema_change | 6 | 100.0% (6/6) | 0.0% (0/6) |
| late_or_missing | 6 | 100.0% (6/6) | 0.0% (0/6) |
| duplicate_keys | 6 | 100.0% (6/6) | 0.0% (0/6) |
| null_spike | 6 | 100.0% (6/6) | 0.0% (0/6) |
| distribution_shift | 6 | 66.7% (4/6) | 0.0% (0/6) |
| model_logic_bug | 6 | 83.3% (5/6) | 0.0% (0/6) |
| broken_join | 6 | 100.0% (6/6) | 0.0% (0/6) |
| timezone_error | 6 | 66.7% (4/6) | 0.0% (0/6) |
| all faulted | 48 | 89.6% (43/48) | 0.0% (0/48) |
Without the tests (every proposed cause ranked) top-1 would be 39.6% (19/48).
False alarms on clean runs: 0.0% (0/30). Five methods on the same 48 dev incidents. Teal is the share of root causes named exactly. Grey is the share named confidently and wrongly.
- dbt testsalert only
- dbt tests + guessalert plus naive cause
- Raw logs → LLMlocal qwen2.5 7B
- faultline monitorsanomalies, ranked upstream-first
- faultline triagemonitors + tested causes
The monitors alone already detect 89.6% of incidents, but they point at the wrong place a third of the time. Testing every proposed cause removes all of those wrong answers. The 5 incidents triage misses are ones the monitors never flagged: two distribution shifts, two timezone errors and one logic bug.
A one-time check on the train split gave a similar picture: 85 of 96 right at top-1 (88.5%), 3 wrong. Both results reproduce byte for byte, and a test checks that they still do.
Sources: triage, monitors, raw_log_llm, dbt_tests, dbt_tests_guess, train/triage.
08What it refuses to claim
A held-out score. Dev is where the method was developed, so 89.6% is an optimistic number. The held-out jira split will be run once, when the method is final. That number goes here, good or bad.
That abstaining works. "Not confirmed" never fired on dev or train. The behaviour exists in code but the benchmark has not tested it yet.
That the model helps. The LLM re-ranker is not built. Every number above is from deterministic code.
09What broke
A green build with empty tables
The first Stripe build passed 105 of 105 tests while every staging table was empty. dbt had read the sources before the seed data loaded. Now seeds load first, and a test fails if any model is empty.
The answers were in the logs
The saved git history still carried the injector's own commit messages, which described each fault. Any method could have read the answer. They are now removed at load time, and a leak test runs on every dev evaluation.
Realistic noise broke clean runs
Dropping a few percent of rows to simulate real data left orders without payments and failed tests on runs that were meant to be clean. Clean runs now vary only by 2% noise on numbers. The cost: volume false alarms became an easy test. That is listed as a known weakness.
10Still open
- Ablations: no tests, no lineage, no model. What each layer is actually worth.
- The local LLM re-ranker, run on my own MacBook.
- One run on the sealed held-out split.
- Incidents where "not confirmed" is the right answer, so abstaining gets tested.
- court_cases is a synthetic project I wrote. Real public court data comes after a licence check.
- Drafting fixes as pull requests (milestone 7) and a read-only MCP server (milestone 8).
11Reproduce it
Every number on this page comes from these commands, run in the repository.
git clone https://github.com/omkar-k0/faultline && cd faultline
make setup # uv env, install, build the 4 dbt test projects
make check # ruff, mypy --strict, dbt build, pytest
make fetch-benchmark # loads the train and dev incidents
make eval-dev # scores every method on dev, writes eval/results/dev/ | File | What it holds |
|---|---|
| eval/results/dev/triage.md | The 89.6% table, per fault type, with config hash |
| eval/results/dev/ablations.md | Triage without tests and without lineage |
| benchmark/manifest.json | The 192 incidents and 70 clean runs in release benchmark-v1 |
| docs/EVAL.md | How top-1, top-3 and "not confirmed" are counted |
The held-out split ships sealed and stays unopened until the method is frozen.