Integrity

What convergence means here

Every scenario asserts three things. After any sequence of edits, renames, deletes, disconnects, crashes and restarts on any number of participants, all participants hold the same tree. That tree is one of the outcomes the scenario allows: where a winner depends on timing, any valid winner may win, never a mix of two. And no document body is lost to a merge: the loser of a name collision survives as a conflict copy, and only an explicit delete removes a document, per Conflict behavior.

How it is checked

Property tests state the algebra the merge depends on and check it over generated inputs. The clock order is total. Merging lifecycle records is commutative, associative and idempotent, which is what lets machines that saw the same records in a different order, in a different grouping, or more than once still agree. Folding a set of signals is order independent and idempotent.

A simulation crate runs many peers in one process on a virtual clock and a virtual network. Links drop, reorder and delay; the network partitions and heals; peers crash and restart from what they persisted. A seeded generator drives it, so a failing run replays from its seed. A relay-mediated variant routes everything through a simulated relay and adds disconnects and reconnects.

Closer to the real thing, a real daemon session runs against the real relay in one process on a paused clock, with the test choosing which side moves next. Ordering questions that would otherwise be sampled over a socket are enumerated instead: a peer's change is placed at every step of another machine's startup, and convergence is asserted from each one.

Above that, the real kutl and kutl-relay binaries, and the agent interface over standard input and output, are driven through the paths a user takes: install, join, sync, go offline, come back, stop. Each journey has hard assertions on what the user sees and what lands on disk.

Scripted scenarios

kutlhub's end-to-end suite, which is not public, adds scripted scenarios that simulate real problems against real daemons and a real relay: concurrent edits, renames and deletes of one document; two documents landing on one path; work done offline and brought back; a daemon killed mid-burst and restarted; thousands of files at once; the same operations replayed under skewed clocks; the relay noticing when two replicas silently disagree. The slow ones run after any change to the relay or the daemon rather than on every commit.

Rate checks

Some defects show up in only a few runs out of a hundred. A single green run is weak evidence about one of those, and equally weak evidence that a fix worked. A convergence fix is gated as a rate instead: the scenario runs a hundred times with the fix and a hundred times without, on the same machine in the same session, with the baseline against itself to set the noise floor. A fix ships when the difference between the two arms is larger than that floor. The runs are recorded so a later release can be read against them.

Not covered

Everything runs on one machine over loopback; the simulation models the network, but no test crosses a real one. No scenario with real daemons has more than three; larger peer counts run only in the simulation. The relay is trusted: nothing tests a relay that lies. And the property tests sample their inputs rather than exhaust them: they generate cases and check those, which is weaker than showing no bad case exists.