Green dashboards, scheduled runs, zero alerts —
and this AI pipeline sent zero emails
This is where it started on my own business, and the mistake I made before I started doing FDE work. This page is the delivery record: how to find that a successful execution is not the same as a business outcome, how to stop the bleeding in three places no dashboard could show you, and how the same judgement transfers to any AI workflow in your factory.
Boundaries first: what can be said, what cannot
Three tiers, kept apart. Mixing them is over-selling — and over-selling once costs you the only thing this page is for.
Sayable — with receipts
- Duplicate billing and the ungated lookup are gone: one full run, balance 22 → 22, zero consumed
- Multi-lead insert fixed: two leads replayed, each handled on its own — one created, one not duplicated
- Paid-lead recovery went from 1/2 to 2/2
- Full production run measured: 140 seconds, 5 billable calls, all 4 stranded records recovered, 4 leads scored 55 / 45 / 45 / 72
Not done — not sayable
- The three downstream pieces (cold-email generation, reply triage, unsubscribe handling) are still inactive
- No real email has been sent — zero, cumulative
- No conversion has happened: this pilot has still never produced a real customer
Undecided — not sayable
- The scoring threshold (80) and the observed scores (best lead: 72) do not line up; the passing branch still has no execution receipt
The starting point: green board, zero output
Three days after go-live I reconciled stage by stage — not by looking at execution status, but by counting items at every stage. When the counts disagree, a green light means nothing.
15 candidates in three days → 13 rejected by internal qualification → only 2 passed
Only one real lead reached the CRM, even though two had email addresses
Downstream: 0 leads, 0 emails sent — the whole chain produced zero outbound messages
Pay-per-call balance: half the credit burned in three days, and the bill never says where
A successful execution + a real invoice + an empty downstream — all three true at the same time.
This is the hardest class of failure in an AI pipeline: it does not error. It simply never finishes the job. The log says success, the board is green, and no email ever went out and no lead ever entered your system.
Three obvious causes — all refuted by evidence
This is the most valuable section of the record. Every explanation offered at kickoff turned out to be wrong — and proving that was also the fastest way forward.
- REFUTED 1
"The threshold is too strict — it filtered everything out"
Refuted. Every one of the 13 rejections carried a reason, and the external scorer gave the single surviving lead 55 with an explanation — both gates were working exactly as designed.
- REFUTED 2
"Execution says success, so the chain is healthy"
Refuted. Two paid leads, one CRM record. Every "successful" run also triggered a billed lookup.
- REFUTED 3
"What we deployed is the file in the repo"
Refuted. All three defects came from assuming the deployed configuration was the adapted one. The repo file was never what ran.
An empty downstream does not mean the upstream blocked everything.
When results look wrong, prove each obvious explanation right or wrong before inventing a new one. That single step narrows the search from "tuning" to "architecture" — otherwise you burn a week on thresholds, prompts and model parameters.
Three defects, all invisible to the dashboard
They were not in the repository, not in the architecture diagram, not on any monitoring panel. They existed only in the deployed state.
A billed call with no gate
- Where
- A second email-lookup chain in the orchestration layer, sitting before the CRM insert and the scoring step
- Cost
- Leads whose emails were already looked up were looked up — and billed — again
- Why nobody noticed
- The execution log looks perfect; you find it when the invoice arrives
The discovery request lost its "only new leads" flag
- Where
- The request body of the discovery node (changed at deploy time; the repo copy was correct)
- Cost
- Every scheduled run returned the same batch of leads, billing again each time
- Why nobody noticed
- Every run returns results, so the repetition is invisible
Single-item extraction
- Where
- The contact-creation node took only the first item
- Cost
- A run with several leads inserted only the first one
- Why nobody noticed
- Execution succeeds and the second lead disappears without a trace
Verification: claimed is not accepted
After the fix the system entered a safe idle — with the upstream repaired, the downstream was never reached and every green light became meaningless. The answer is not to wait; it is to manufacture a sample.
- No duplicate billingOne full run in 2m35s, status success, balance 22 → 22, zero consumed, and no lookup call anywhere in the run
- Leads handled individuallyOne created with email, title and timestamp intact; the other not duplicated, its original timestamp unchanged
- Real production run140 seconds, 5 billable calls, all 4 stranded records recovered (insert count 8 → 12)
- Scores reproducibleThe 4 leads scored 55 / 45 / 45 / 72 — none crossed the threshold of 80
- Fixture removedThe replay fixture was reset and the workflow is still running
- How the fix landed19 atomic operations through the orchestration API, all-or-nothing rollback; the old chain was disconnected and labelled, not deleted
A result without a receipt does not get called delivered.
The fix was booked as a claim until a replay produced a receipt, at which point it was promoted to accepted. Receipts are not paperwork — they are the only answer to "how do you know you fixed it", and the only evidence you and your boss will not argue about.
The replay also flushed out two real defects that the idle path had been hiding (values read from the wrong wrapper, and a cross-node reference quietly downgraded) — both fixed. That is the cost of the rule that an idle path is a disguise for defects: until you make it run once, it keeps hiding.
How this transfers to your factory
It does not depend on my stack. Any AI workflow — from classifying inbound enquiries to preserving a master technician's know-how — runs the same checklist.
- 01
Success is not completion
"Execution succeeded" is the signal most likely to lie to you. What you verify is conservation of items — does each stage take in what it should and pass on what it should — not whether the light is green.
- 02
Falsify the obvious causes first
Three commonsense explanations were all refuted before the real architectural defect could be isolated. The time this saves far exceeds what it costs.
- 03
The deployed state is the truth
The config in the repo and the architecture diagram are not evidence. All three defects here came from assuming the deployed version was the adapted one.
- 04
Billed calls need a gate — and the gate must be read off the deployed graph
A well-designed internal gate can sit right next to an ungated side channel. Both were running at the same time.
- 05
Leave a traceable before/after
The old chain was disconnected and labelled rather than deleted: it is both the audit trail and the physical evidence of the comparison.
- 06
An idle path hides defects
Once the upstream is fixed, the system may never reach the downstream simply because there is no data — and then every green light is meaningless. Change a data path and you must force it down that path with a fixture, or acceptance is theatre.
The end of a delivery is not a pipeline that runs
It is a system that runs faster next time
Reusable knowledge does not move by itself — experience scattered across chats and throwaway scripts is gone the moment the session ends. Here the validated method was written down as reusable assets (delivery playbook, evaluation rubric, executable skills), with a two-track memory of process state and knowledge state: the method lives in the system, not in one person's head.
And that is why this is the work
What a client buys is not a one-off bug fix — it is every new workflow running under the same discipline afterwards: reconcile first, change the path second, keep the receipt, and build the asset.
Common questions about this record
Is this a client case study?
No. It is a delivery record from a self-run pilot — my own business was the first customer, and the person who accepted the work is the person who did it. That is not client sign-off. It demonstrates method and judgement, not business outcomes.
Can you guarantee this will not happen in my project?
We cannot guarantee it will not happen, but we can guarantee it gets found by the same method: reconcile every stage, verify conservation of items, and require an execution receipt. Everything on this page carries a receipt; claims without receipts do not get called delivered.
How long did this take?
On the first day of delivery we wrote no new code. We found and stopped the bleeding in three defects so that "execution succeeded" meant "the business actually happened". Verification, replay and asset-building came after.
How do I know my own AI pipeline is idling?
Count three things: items in versus items out at each stage, billing consumed versus actual output produced, and whether the far end actually happened (what was sent, what landed in which system). If those three do not reconcile, the pipeline is idling.
Why is this useful to me?
It does not depend on my stack. Any AI workflow — from classifying inbound enquiries to preserving a master technician's know-how — runs the same checklist: conservation at every stage, a gate on every billed call, a receipt behind every "I fixed it".
Is your AI workflow running — or idling?
A free 30-minute call. Describe the most time-consuming step; I will first tell you whether it is worth touching — and if it is not, I will say so plainly.