Your Pipeline Ran for Three Days. The Output Was Trash in Hour One.

You know this failure. The workflow finishes Sunday night. Monday morning the last step crashes with KeyError: 'ref' under forty frames of traceback that name everything except the problem. The real defect happened Thursday morning: step 3 of 9 wrote a CSV whose header says reference instead of ref. Steps 4 through 8 consumed that file without a complaint, because nothing in the stack checks content: the file existed, the bytes were CSV-shaped, and the next process started. The stack’s position on your CSV’s contents is that they are between you and your CSV.

The instinctive fix is “validate more”: wrapper scripts, schema checks in CI, synthetic-data smoke runs. These help, and they don’t fix it, because the failure was never a missing check. It was a check scheduled for the wrong time. Static pipeline definitions validate structure at submit time. Content gets validated when a step happens to look at it, if it looks. The space between “this graph typechecks” and “these bytes satisfy that graph” is where the weekend went.

Pointy is my answer to that gap. The design in one sentence: one program, two checks, and the checks are staged so each runs at the first moment its inputs exist.

Check one: the graph is a typed program

A step’s promise in pointy is a term in the type system, not documentation:

struct Alignment
  (library : @fulfill Library)
  (dataSource : @fulfill Directory)
  (extraArgs : String = "")
  : Directory < /Aligned.out.bam, /SJ.out.tab >

An alignment step takes two inputs that must be previous steps (that’s what @fulfill marks) and commits to producing a directory containing at least those two files. struct Library separately commits that its /lib.csv is a CSV with ref and seq columns of non-null text. Here is the whole contract for that file:

struct Library : Directory
  < lib @ /lib.csv : Csv <ref : Text NonNull, seq : Text NonNull>
  , /lib.fa
  >
  where (size lib <= 1048576)

A Library is a directory that must contain lib.csv and lib.fa. The Csv <…> tail reaches inside the CSV: two named columns, both text, never null. The where caps the file at one megabyte. No wrapper script, no separate schema file: the promise lives in the type.

Wiring the graph is typechecking: a consumer’s pattern has to unify with some producer’s promise. Two consequences fall out of this being unification rather than configuration.

First, invariants you’d normally enforce with code review come free. Writing alignments : List (@fulfill (Alignment library _ _)) binds library from the first alignment’s producer and requires every later element to match it. All alignments share one reference genome, not because there’s a “same-library rule” anywhere in the codebase, but because that’s what unification means. There is no such rule; there couldn’t be a simpler one.

Concretely, here is a consumer that feeds a whole list of alignments into one histogram:

struct Histogram
  (alignments : List (@fulfill (Alignment library _ _)))
  : Directory < /counts.csv >

Two rules of pattern syntax do all the work: a lowercase word (library) is a variable that gets bound, and writing the same variable twice means the bound values must be equal; an underscore marks a position we don’t care about. So the first alignment hands its library to library, and every later element must carry the same one. Point two alignments at different libraries and the graph check refuses (lightly trimmed for print):

error: step 5: claim parameter "alignments": capture "library" must
resolve to the same producer step in every element

Second, and less obvious: a consumer can demand a file the producer never promised, and that is not an immediate error. The obligation survives to the second check, against the real directory at build time. Only an outright contradiction (the producer promised this path as a symlink and the consumer requires a file) fails statically. The whole policy is three lines in one function.

In practice: an over-demanding consumer fails the first time the real directory exists, with the error naming what it wanted, not silently, three days in.

Check two: the build discharges the promises

When the alignment finishes, a small sidecar build re-checks what the step actually produced against everything it promised, using only the output directory and the certificates for every producer the step read. Format-specific scanners open the real bytes and try to discharge the leaf claims. Does lib.csv really have a ref column? Is it null-free? I exercised this while writing: a CSV missing a promised column is rejected at scan time, and so is a column whose values parse as integers when the claim says text. File existence is not a fact. Column content satisfying the claim is.

Here is that failure verbatim. A CSV with a reference column where the promise requires ref:

$ pointy fulfill --extension csv.pointy --output out lib.pointy prepare
error: csv.pointy:24:5: constraint failed: contains(columns, observed) (returned false)
  claim /lib.csv: promised column `ref` was not observed

The scanner’s column check (contains(columns, observed), declared at line 24 of the csv extension) returns false; the failure names the offending member and the missing column, and the step’s own build fails on the spot. The forty-frame alternative names neither the member nor the column, and arrives Monday.

Failure here arrives with a source span, lib.pointy:6:12: constraint failed: size(lib) <= 1048576, pointing at the where-clause that declared it, not as a downstream parse error two days later.

And the failure is structural, not advisory: a step cannot start at all until every producer it reads is certified. Underneath, that is ordinary build wiring (consumer steps list producer certificate derivations as build dependencies), so rewiring the graph into a shape where certificates don’t line up makes nix build refuse before anything runs.

The staging mechanism is one clause

The compiler generates two evidence relations over the same proof obligations, from your struct declarations. One walks the obligations at graph-check time and stops at anything the author marked for the build stage. The other walks the same tree at build time and must descend into everything. Here is the entire difference:

prop-prepares   Bs Bs   (prop-fulfill A).
prop-evidences  Bs0 Bs1 (prop-fulfill A) :- prop-evidences Bs0 Bs1 A.

There is no flag threaded through a scheduler, no second implementation to drift. The two stages are two readings of one term, parametric in which facts exist when you ask. They can never disagree about an unstaged claim, because they share every clause that matters.

In practice: you write each claim once. pointy check reads it as a plan; the build reads it against real bytes. There is no switch to forget to flip.

What this changes on Monday morning

The reference/ref mismatch now fails step 3’s build on Thursday, with a message naming the struct and the column. Your current scheduler calls this pattern “Tuesday”. The later class of trap, a consumer quietly training on a malformed artifact, is structurally gone: the consumer can’t build before the producer’s certificate exists and matches the store path it reads.

The limit, kept in the open

This verifies claims, not correctness. A FASTA can have non-null sequence text in every row and still be the wrong genome. Scanners are trusted code; the docs say plainly that the certifier does not certify scanner correctness. What’s eliminated is the whole taxonomic family of plumbing failure (wrong columns, missing members, nulls where non-nulls were assumed, silently oversized inputs), every one of which now terminates in a build error that names the source span.

The goal was never to prove the pipeline. It was to move information from the moment of damage to the moment of definition. Three-day failures are timing bugs: the fact was knowable on Thursday morning, and staged typechecking just makes asking it then mechanical.