Your Pipeline Ran for Three Days. The Output Was Trash in Hour One.

You know this failure. The workflow is still running Sunday night. Monday morning the last step crashes with KeyError: 'ref' under forty frames of traceback that name everything except the problem. The real defect happened Thursday morning: step 3 of 9 wrote a CSV whose header says reference instead of ref. Steps 4 through 8 consumed that file without a complaint, because nothing in the stack checks content: the file existed, the bytes were CSV-shaped, and the next process started. The stack’s position on your CSV’s contents is that they are between you and your CSV.

The instinctive fix is “validate more”: wrapper scripts, schema checks in CI, synthetic-data smoke runs. These help, but they don’t fix it, because the failure was never a missing check. It was a check scheduled for the wrong time. Static pipeline definitions validate structure at submit time. Content gets validated when a step happens to look at it, if it looks. The space between “this graph typechecks” and “these bytes satisfy that graph” is where the weekend went.

Pointy is my answer to that gap: one program, two checks, the checks staged so each runs at the first moment its inputs exist.

Check one: the graph is a typed program

A step’s promise in pointy is a term in the type system, not documentation:

struct Alignment
  (library : @fulfill Library)
  (dataSource : @fulfill Directory)
  (extraArgs : String = "")
  : Directory < /Aligned.out.bam, /SJ.out.tab >

An alignment step takes two inputs that must come from previous steps, which is what @fulfill marks, and commits to producing a directory containing at least those two files. A Library commits separately that its /lib.csv is a CSV with ref and seq columns of non-null text:

struct Library : Directory
  < lib @ /lib.csv : Csv <ref : Text NonNull, seq : Text NonNull>
  , /lib.fa
  >
  where (size lib <= 1048576)

The Csv <…> tail reaches inside the CSV itself: two named columns, both text, never null; the where caps the file at one megabyte. No wrapper script, no separate schema file: the promise lives in the type.

Wiring the graph is typechecking: a consumer’s pattern has to unify with some producer’s promise, and because that wiring is unification rather than configuration, invariants you’d normally enforce with code review come free. Writing alignments : List (@fulfill (Alignment library _ _)) binds library from the first alignment’s producer and requires every later element to match it. All alignments share one reference genome, not because there’s a “same-library rule” anywhere in the codebase, but because that’s what unification means. There is no such rule; there couldn’t be a simpler one.

A consumer feeding a whole list of alignments into one histogram:

struct Histogram
  (alignments : List (@fulfill (Alignment library _ _)))
  : Directory < /counts.csv >

In a pattern, a lowercase word binds, the same word written twice requires the two bound values to be equal, and an underscore discards the position. The first alignment hands its library to library; every later element must carry the same one. Point two alignments at different libraries and the graph check refuses (lightly trimmed for print):

error: step 5: claim parameter "alignments": capture "library" must
resolve to the same producer step in every element

A quieter property runs the other way: a consumer can demand a file the producer never promised, and that is not an immediate error. The obligation survives to the second check, against the real directory at build time; only an outright contradiction, the producer declaring this path a symlink while the consumer requires a file, fails statically. The entire policy is a three-line match in one function.

An over-demanding consumer fails before it runs, the first moment everything it reads exists, with the error naming what it wanted instead of a crash three days in.

Check two: the build discharges the promises

When the alignment finishes, a small sidecar build re-checks what the step actually produced against everything it promised, using only the output directory and the certificates for every producer the step read. Format-specific scanners open the real bytes and try to discharge the leaf claims. Does lib.csv really have a ref column? Is it null-free? I exercised this while writing: a CSV missing a promised column is rejected at scan time, and so is a column whose values parse as integers when the claim says text. File existence is not a fact. Column content satisfying the claim is.

Verbatim, for a CSV with a reference column where the promise requires ref:

$ pointy fulfill --extension csv.pointy --output out lib.pointy prepare
error: csv.pointy:24:5: constraint failed: contains(columns, observed) (returned false)
  claim /lib.csv: promised column `ref` was not observed

The column check (contains(columns, observed), line 24 of the csv extension) returned false, so the step’s build stops there, with the message naming the member and the missing column. The forty-frame alternative names neither, and arrives Monday.

A violated where fails the same way: lib.pointy:6:12: constraint failed: size(lib) <= 1048576 points at the clause that declared it, not at a downstream parse error two days later.

The failure is structural, not advisory: a step cannot start until every producer it reads is certified. Underneath, that is ordinary build wiring; consumer steps list producer certificate derivations as build dependencies. A graph rewired into a shape where certificates don’t line up is refused by nix build before the consumer runs.

The staging mechanism is one clause

The compiler generates two evidence relations over the same proof obligations, from your struct declarations. One walks the obligations at graph-check time and stops at anything the author marked for the build stage. The other walks the same tree at build time and descends into everything. The entire difference:

prop-prepares   Bs Bs   (prop-fulfill A).
prop-evidences  Bs0 Bs1 (prop-fulfill A) :- prop-evidences Bs0 Bs1 A.

There is no flag threaded through a scheduler and no second implementation to drift. The two stages are two readings of one term, parametric in which facts exist when you ask; they can never disagree about an unstaged claim, because they share every clause that matters.

Each claim is written once. pointy check reads it as a plan, the build reads it against real bytes, and there is no switch to forget to flip.

What this changes on Monday morning

The reference/ref mismatch now fails step 3’s build on Thursday, with a message naming the struct and the column. The later class of trap, a consumer quietly training on a malformed artifact, is structurally gone: the consumer can’t build before the producer’s certificate exists and matches the store path it reads.

Where this leads

The whole design comes down to where the knowledge lives: with the producer that wrote the bytes, or with the reader that needs them. When the producer promises the shape, as Library does for its ref column, the failure happens at the moment the damage is done. When only a reader requires it, the check still runs before that reader starts, but everything in between has already run. Same machinery, same error message; the only difference is when the machinery gets its first look.

The natural evolution of the system is to remove that difference. The compiler already knows every reader and everything each one demands, so a producer’s certificate could answer for all of it: not just “this step kept its promises” but “everyone downstream signed off,” discharged in the producer’s own build. The price is coupling. The certificate becomes a function of its reader set, a new reader re-certifies the producer and rebuilds everything downstream, and an artifact is only as reusable as the demands it was certified against. How much of that price a pipeline should pay is a design question, not a technical one.

One way to read this post is as a single number: the time between when bad bytes are written and when anyone finds out. An unchecked scheduler measures it in days. The staged checks above measure it in minutes after the guilty step finishes. A reader-aware certificate would measure it in the producing step itself, wherever in the graph the requirement happens to be written. That version doesn’t exist yet. It’s the one I’m building.