❯ Testing
The test suite is not evidence of the product. It is the product.
A tool whose pitch is control cannot make a claim it has not verified. Guards decide what may move and the repository records what moved — this is why you can believe either.
The targets below are targets. They come from the design, not from a coverage run, and they will be printed beside the measured actuals once the figures are generated in CI rather than typed. Publishing a target as though it were a result is the exact failure this page exists to disprove.
12 invariants, 12 named tests
Each safety rule is an entry in a catalogue that names the test holding it, and a test over the catalogue asserts that every named test exists and passes. The rule stops being prose: delete the enforcement and the build tells you which invariant just lost its cover.
An entry may be marked pending while its code is still being written — and the marker is checked in both directions. Leaving it on after the test lands fails just as loudly as omitting it before, so a marker cannot outlive the work it stands for.
| Invariant | What it says | Asserted by |
|---|---|---|
| PROTECTED_NEEDS_FORCESPEC §5.4 | a protected environment refuses every upward write without --force | policy: a protected environment refuses an upward write without --force |
| REFUSAL_SURVIVES_DRY_RUNSPEC §5.4 / §19.1 | the refusal runs under --dry-run too; the consent does not | policy: a protected environment refuses under --dry-run as well |
| BACKUP_BEFORE_UPWARDSPEC §5.4 | no remote write proceeds without a restore point, or an explicit waiver | policy: an upward write needs a restore point or an explicit waiver |
| BACKUP_CAPABILITY_REFUSES§9.2 | a provider that cannot back up refuses the upward write rather than skipping it | policy: a provider that cannot back up refuses rather than skipping |
| UPWARD_HAS_A_SOURCEP2 / F-043 | an upward effect with no repo source is refused | policy: an upward effect with no repo source is refused |
| WRITE_SCOPE_IS_PER_EFFECT§13 / §19.3 | direction lives on the effect, never on the command | policy: direction is read from the effect, not the command |
| TRACKED_NEVER_OVERWRITTENSPEC §5.8 | a downward fetch never overwrites a path listed in tracked — that is your source, not the remote's | policy: a fetch never overwrites a tracked path |
| NEVER_RAW_SQLCLAUDE.md rule 6 | URL replacement goes through wp search-replace, never raw SQL | policy: a URL rewrite must use search-replace, never raw SQL |
| SNAPSHOT_ABORTS_PULLSPEC §5.3 / §19.5 | a failed pre-pull snapshot aborts the pull — the one step of seven that is not best-effort | ops/pull: a failed pre-pull snapshot aborts the pull |
| MAIL_GUARD_ALWAYSCLAUDE.md rule 5 / §19.4 | every pull installs the mail guard, on every terminating path, even with scrubbing disabled | ops/pull: the mail guard is installed on every terminating path |
| REFUSAL_HAS_NO_EFFECTSSPEC §5.4 | a refused push runs no hook, takes no backup and transfers nothing | ops/push: a refused push has no side effects at all |
| VERIFY_BEFORE_NEXTSPEC §5.4 / §19.2 | each transfer is verified as it lands; a failed digest stops the fold | ops/push: a failed digest stops the fold before the next transfer |
The build fails if any named test disappears. These are not documentation of intent — they are the tests, indexed by the rule they defend.
Four tiers, four different jobs
domain/≥ 95%
None — plain values, and real temp directories where a planner stats.
Planning, precedence, parsing, canonical form.
tools/≥ 90%
None — exact argv assertions, including providers and builders.
The highest-value tests in the project: a wrong --app, a wrong host, a wrong path. That is the class of bug that damages a site, and only argv assertions catch it.
ops/≥ 85%
A scripted session.
Step ordering, guard placement, dry-run behaviour.
cli/≥ 70%
Manifest-driven conformance.
Flag plumbing, exit codes, envelope shape.
Every seam is tested twice
One conformance suite runs against every implementation of a seam: cloudron, ssh and local for hosts; bricks, none and elementor for builders. A behaviour that only holds for the first implementation fails immediately — which is the entire reason a second implementation exists.
It was verified by breaking it deliberately: dropping a fix hint from a command, and letting a host decide its own read/write intent instead of honouring the caller's, each fail the run.
Three structural tests
- The layering check
- Upward imports fail. Sibling imports stay legal, because a rule against those invents a
shared/directory within a month. - The invariant catalogue check
- Every entry names a test that exists and passes.
- The envelope-shape check
--jsonkeeps the documented shape, so anything parsing it can rely on the contract rather than on the current release.
No test needs a live host or a running Docker daemon. The subprocess boundary is the seam, and it is scripted.