metatheca — Specification

version 0.2

This document specifies the metatheca protocol: a path-and-fact layer over a syntheca cella. metatheca maps human-meaningful paths onto stable opaque entry IDs, records structured metadata as content-addressed facts, captures the cumulative fact set per moment as state blobs forming a single-track chain, and exposes the current state through a single mutable head pointer (an apotheca pinax on the same cella).

The specification is language-agnostic. The reference implementation is in Rust (../metatheca-rs/).

For motivation, design choices, and the relationship to the substrates, see README.md. For the substrate primitives this document builds on, see ../../syntheca/spec/SPEC.md and ../../apotheca/spec/SPEC.md. This document covers what sits above those: vault layout, identifiers, fact wire format, core fact namespaces, state-chain semantics, head-pinax protocol, projection rules, and the CLI surface.

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY in this document are to be interpreted as described in RFC 2119.

1 Scope and terminology

This document specifies metatheca Phase 1: a single local vault on top of a single local syntheca cella, with deposit-only durability (no blob deletion, no garbage collection) and no public hook trait. Phase 1 is the v0.1 release surface; deferred items are listed in §10–§11.

Terminology not redefined here is inherited from syntheca SPEC §1 and apotheca SPEC §1. In particular: cella, depositum, pinax, hash (BLAKE3), digest (SHA-256), name.

1.1 Vault

A vault is the unit metatheca owns. A vault bundles:

A vault is identified at runtime by a filesystem path, the vault root (§7). The library MUST take the vault root as a caller-supplied parameter; it MUST NOT bake in a default. The CLI MAY apply a default for shell convenience (§8).

1.2 Entry

An entry is the stable opaque identity of a thing in the vault. An entry is referred to by an entry ID (§2.1). Entries exist independently of paths: the same entry MAY have zero, one, or many paths over its lifetime.

1.3 Path

A path is a UTF-8 string naming an entry from the user’s perspective. Paths are mutable references: renaming a path emits new facts, but the entry it referred to is unchanged. Multiple paths MAY refer to the same entry simultaneously.

Path constraints:

1.4 Fact

A fact is a structured record about an entry. Facts are stored as content-addressed blobs in the cella’s depositum namespace (§3.2). Facts are the source of truth for everything mutable in the vault; the SQLite index (§7) is a regenerable projection of the fact set.

Facts are namespaced. Core facts (kinds beginning core/) are specified in §4 and emitted by metatheca itself. Extension facts are out of Phase 1 scope (§10).

1.5 State

A state is a moment in the vault’s history. Each state is materialized as a content-addressed state blob (§3.3) capturing the new facts added since the prior state and a back-pointer to that prior state. The set of states linked through these back-pointers is the state chain.

The genesis state is the unique state with previous = null, created by init (§5.1).

1.6 Head

The head is the single mutable pointer identifying the current state of a vault. It is realized as an apotheca pinax (§6) on the vault’s cella, holding the BLAKE3 hash of the current state blob.

Head is plumbing. Library and CLI surfaces use the noun state; the word head appears only in the head-pinax subsection of this document and in implementation internals.

1.7 State reference

A state reference (or stateref) is any of the following, resolving to a unique state in the chain (§5.3):

A stateref MUST resolve to exactly one state or fail. Ambiguous hash prefixes MUST surface an error listing the matches.

1.8 Index

The index is a SQLite database file storing the projected current state derived from replaying all facts in the chain up to a chosen state. The index is a regenerable cache; it is not a source of truth and its schema is not part of this specification (§7).

2 Identifiers

2.1 Entry ID

An entry ID is a UUID v7 (RFC 9562 §5.7) — 128 bits, with the upper 48 bits encoding a Unix-epoch millisecond timestamp.

The canonical wire form on the JSON surface (facts, state blobs) is the RFC 4122 string form: 32 lowercase hex digits with hyphens at positions 8, 13, 18, 23 (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx).

Entry IDs are generated by metatheca on entry creation. Implementations MUST use a UUID v7 generator that satisfies RFC 9562 §6.2 (monotonic within a millisecond) for entries created in the same vault process.

2.2 Hashes

Content-addressing throughout metatheca uses syntheca’s BLAKE3 hash (syntheca SPEC §1.1, §1.2). Wire form: 64 lowercase hex digits.

2.3 Time

All timestamps are nanoseconds since the Unix epoch as a signed 64-bit integer.

State created_at_ns is monotonic per chain, not merely per process: a state’s created_at_ns MUST be strictly greater than its previous state’s created_at_ns. A committing writer therefore clamps against the parent state it is building on — if the wall clock reads at or before the parent’s created_at_ns (clock step-back, another writer with a faster clock, a vault moved across machines), the writer MUST use parent.created_at_ns + 1 and SHOULD log the event. Per-chain monotonicity is what makes instant staterefs (§1.7, §5.3) and time-windowed log filters (§8.10) exact rather than chain-order-dependent.

3 Wire formats

3.1 Canonical JSON

Fact blobs and state blobs are serialized as canonical JSON per RFC 8785 (JSON Canonicalization Scheme, JCS). In summary:

Deviation from strict JCS (normative): every integer value in a fact or state blob — the core fields (version, size, created_at_ns, the core/fs-metadata times and mode) and any integer appearing in a fact body, including extension-namespace bodies — is serialized as an exact decimal integer: no exponent, no fraction, no leading zeros, - sign only. This holds even when the value exceeds the range IEEE-754 binary64 represents exactly (|x| > 2^53). Strict RFC 8785 serializes numbers through binary64 (§3.2.2.3), which would silently round a 64-bit nanosecond timestamp. Implementations MUST NOT route integer values through a binary64 representation. Off-the-shelf JCS libraries that implement strict §3.2.2.3 are therefore not conformant for this protocol without an exact-integer path.

Integer values MUST lie in [-(2^63), 2^64) (i.e. representable as a signed or unsigned 64-bit integer). Numbers outside that range, and non-integer numbers (fractions, exponents), are Malformed — writers MUST NOT produce them, and readers MUST reject fact and state blobs containing them, including within fact bodies.

Implementations MUST canonicalize before hashing and storing. Two facts or state blobs are equal iff their canonical bytes are equal, and equal canonical bytes MUST hash to equal BLAKE3 digests.

3.2 Fact blob

A fact blob is a canonical-JSON object with the following fields:

{
  "type": "fact",
  "version": 1,
  "kind": "<namespace>/<name>",
  "entry": "<uuid-v7>",
  "body": { /* kind-specific, defined by the kind */ }
}

Field constraints:

A fact’s identity is the BLAKE3 hash of its canonical-JSON bytes.

3.3 State blob

A state blob is a canonical-JSON object:

{
  "type": "state",
  "version": 1,
  "previous": "<blake3-hex>" | null,
  "added_facts": ["<blake3-hex>", "..."],
  "created_at_ns": 1730000000000000000
}

Field constraints:

A state’s identity is the BLAKE3 hash of its canonical-JSON bytes.

The cumulative fact set at state S is the union of added_facts over all states from genesis through S inclusive. Phase 1 implementations compute this by walking the chain.

Note (non-normative): a cycle in the chain is unconstructible under honest content-addressing — a state’s hash covers its previous pointer, so a state cannot name a descendant — and walks therefore terminate. Implementations MAY still guard walks with a seen-set as defence in depth against corrupted or adversarial blob stores.

4 Core fact namespaces

The core/ namespace is reserved for facts emitted by metatheca itself. Phase 1 defines four kinds; their body schemas are normative.

4.1 core/blob-ref

Asserts the content blob backing an entry.

{ "blob": "<blake3-hex>", "size": <integer> }

Emitted at ingest. Re-emitted if the entry’s content is replaced. The projection takes the latest such fact (§5.5) as the current content.

Content blobs are ordinary syntheca deposita; an implementation MAY deposit or read them through syntheca’s streaming forms (syntheca SPEC §2.4, §2.5) with results indistinguishable from the buffered operations — the blob’s identity and this fact’s shape are unaffected by how the bytes arrived.

4.2 core/path

Asserts or retracts the binding of a path to an entry.

{ "path": "<utf8>", "linked": true | false }

The projection (§5.5) treats the latest core/path fact for a given path as authoritative. A path is current iff its latest fact has linked = true. A path may transition between entries by emitting an unlink for the old entry and a link for the new in the same state.

Rebinding. Asserting a path that is currently bound to a different entry rebinds it: the newer fact is latest and therefore authoritative, and the prior entry’s binding is superseded without an explicit retraction. The superseded entry keeps no claim on the path — if it has no other current path it remains reachable only by entry ID (and through historical states via §8.11). A retraction (linked = false) affects only the (path, entry) pair it names: a stale retraction emitted by a former owner does not unbind the path from its current owner. These are the fact-level semantics; the operation surface constrains when rebinding may happen (§8.5, §8.6).

4.3 core/fs-metadata

Records filesystem metadata captured at ingest.

{
  "ctime_ns":      <integer> | null,
  "mtime_ns":      <integer> | null,
  "birthtime_ns":  <integer> | null,
  "mode":          <integer> | null,
  "source_path":   "<utf8>"  | null
}

All fields are nullable to accommodate platforms or sources that do not provide a given field. Times are nanoseconds per §2.3 (drawn from the source filesystem, not the metatheca clock). mode is the POSIX mode bits as an unsigned integer; on platforms without POSIX modes, null.

Emitted at most once per entry, at ingest. Re-emission is permitted (e.g. on a corrected ingest) but the projection takes the latest as authoritative for each field independently.

4.4 core/mime

Records the detected MIME type of an entry’s content.

{ "mime_type": "<utf8>" }

mime_type SHOULD be a registered IANA media type; implementations MAY emit non-registered types when detection cannot do better (e.g. application/octet-stream). The projection takes the latest such fact as authoritative.

5 Operations

A vault exposes the following protocol operations. Each is total: it terminates with one of the outcomes listed for it.

5.1 init

Creates a new vault at a previously-empty filesystem path (§7).

  1. Create the vault directory and the cella subdirectory; open the syntheca cella at the cella subdirectory. (Cella construction is implementation surface, not protocol: the syntheca library takes the root as a caller-supplied parameter — see syntheca SPEC §5.)

  2. Construct the genesis state blob: previous = null, added_facts = [], created_at_ns = now(). Deposit it via syntheca.deposit; let g be the returned hash.

  3. Set the head pinax (§6) to g with expected = None. The precondition MUST hold; if it does not, the vault root is in use and init fails.

  4. Create the SQLite index file and apply the initial schema (§7).

On any failure prior to step 3, the implementation MUST leave no partially-initialized head pinax. Failures after step 3 leave a usable vault whose index is missing or partial; reindex (§5.6) restores it.

5.2 commit

Atomically advances head from the current state to a new state incorporating a set of new facts.

Inputs: a list of fact blobs (canonical-JSON bytes) to add.

  1. For each input fact, compute its canonical-JSON bytes (§3.1) and call syntheca.deposit to obtain the fact hash. Idempotent re-deposit is benign.

  2. Read the current head pinax bytes b via syntheca.get_pinax(name="head"); parse them as the current state hash h per §6.2, and compute d = sha256(b). d is the digest apotheca stores for the head pinax (apotheca SPEC §3.5) — note it is the digest of the 64-octet hex encoding, not of the state blob those bytes name.

  3. Construct the new state blob: previous = h, added_facts = [hashes from step 1, in caller-specified order], created_at_ns = now(). Deposit it via syntheca.deposit; let s be the returned hash.

  4. Call syntheca.set_pinax(name="head", bytes=s_bytes, expected=Some(d)) where s_bytes is the canonical encoding of s per §6.2.

  5. On Ok, bring the index projection up to the new state: apply, in chain order, every state between the previously projected state and the new head — not merely the facts of this commit. (After a CAS retry, the new state’s parent may be a state committed by another writer that this process has never projected; applying only the commit’s own facts would silently and persistently omit those states from the index.) On Conflict, another writer advanced head concurrently; the caller MAY retry by repeating from step 2 with the reported actual digest.

The commit point is step 4. Steps 1–3 are pre-commit and produce only content-addressed deposita that are safe under interruption (orphaned blobs cost storage but do not corrupt the vault).

Implementations SHOULD additionally expose a parent-pinned form of commit taking an expected parent state: it performs steps 1–4 exactly once, failing with Conflict — before step 4 when head is not the given parent, or at step 4 when it stops being — and never rebasing onto a moved head. Callers whose fact batch was derived from the parent state (a read-modify-write, e.g. an allocation against a counter fact) use this form and drive the retry themselves, rebuilding the batch from a fresh read; the internally-retrying form is appropriate only for batches whose meaning is independent of the state they land on.

5.3 Resolve a state reference

Given a stateref (§1.7) and the current head, return the unique state hash it denotes:

Implementations MAY cache stateref → state-hash resolutions for the duration of a process and MAY persist resolutions in the SQLite index to amortize repeated walks; such caches MUST be invalidated when head moves.

5.4 Walk the state chain

Given a starting state, return an iterator yielding states from that state back to genesis by following previous. Implementations MUST fetch each state via syntheca.get and parse it per §3.3. The walk terminates at the genesis state (the unique state with previous = null).

5.5 Project facts to index

The index (§7) is the projection of the cumulative fact set up to a chosen state. The projection rules:

5.6 Reindex

Discard the SQLite index file and rebuild it from a chosen state (default: current head). Equivalent to a fresh init-style index schema followed by §5.5 against the chosen state. Reindex MUST NOT modify any depositum or pinax.

5.7 Fact queries

In addition to the kind-specific tables of §5.5, the projection MUST record the generic fact log: every applied fact of every kind — including kinds the implementation does not otherwise recognize — in chain application order, with the created_at_ns of the introducing state. (§5.5’s permission to skip unrecognized facts applies to kind-specific tables only.)

Three query operations are defined over the log. Each MUST return exactly what a forward replay of the chain up to the projected state would produce:

The operations MUST also be answerable against a historical view (§8.11), where they reflect the viewed state.

This is the substrate half of the fact-index usage pattern: a consumer that stores its own <app>/* namespaces (cuj, dbaiv) reads them back through indexed queries instead of replaying the chain per process, and latest-wins snapshot facts become cheap current-state documents. The wire format is unchanged — the log is a projection of facts that were always there.

6 Head pinax

6.1 Name

The head pinax MUST be stored under the apotheca pinax name head (four ASCII octets: h, e, a, d). This name is fixed by this specification.

The name is a valid apotheca name (apotheca SPEC §4.1): non-empty, length 4, no /, no NUL.

6.2 Bytes

The head pinax bytes MUST be the 64-octet ASCII encoding of the current state’s BLAKE3 hash as 64 lowercase hex digits, with no trailing newline or whitespace. Implementations MUST reject head pinax bytes that do not match this encoding when read.

This is the same encoding syntheca uses for depositum names (syntheca SPEC §1.2), so a head value can be passed directly to syntheca.get after pinax-fetching.

6.3 Compare-and-swap protocol

All updates to head MUST go through syntheca.set_pinax (apotheca SPEC §2.5) with expected set to the SHA-256 digest of the current head pinax bytes — computed either from the bytes passed to the prior successful set_pinax, or by recomputing over the bytes returned by syntheca.get_pinax. This is the digest apotheca stores for the pinax (apotheca SPEC §3.5); it is the digest of the 64-octet hex encoding (§6.2), not of the state blob those bytes name. This makes metatheca’s head advancement linearizable per apotheca SPEC §5.4.

The genesis case (init, §5.1) is the sole exception: it sets head with expected = None, requiring head to be absent.

7 Vault layout

A Phase 1 vault root has the following filesystem layout:

<vault-root>/
├── cella/         — the syntheca cella
├── index.db       — the SQLite projection (regenerable)
└── ext/           — reserved for extensions (optional, see below)

The cella is a complete syntheca cella, stored on disk per the local backend layout (apotheca SPEC §6); metatheca consumes it through the syntheca API and does not reach into apotheca’s on-disk layout directly.

The SQLite schema is implementation-defined and explicitly not part of this specification: it is regenerable from the cella via §5.6, and may evolve between versions without migration ceremony. Implementations MAY include additional vault-root files (configuration, lockfiles, caches); such files MUST NOT be required for vault portability — a vault MUST be openable from cella/ alone, with index.db rebuilt on demand.

The ext/ subtree is reserved for extensions that build derived views over the metatheca cella. Each extension occupies its own subdirectory ext/<extension>/, typically containing its own syntheca cella at ext/<extension>/cella/ and any extension-private auxiliary state. metatheca itself MUST NOT write to ext/; managing the subtree is the responsibility of each extension per its own specification. Because extension state is wholly derived from the metatheca cella, a vault remains portable from cella/ alone: ext/ may be discarded and rebuilt by the extensions’ reindex operations.

8 CLI surface

The reference CLI binary is metatheca with the recommended alias mt. It exposes the protocol against the vault rooted at the current working directory unless overridden with --vault <PATH>.

This section specifies the protocol-bearing subcommands. Diagnostic output formatting is implementation-defined unless otherwise noted.

8.1 mt init

mt init [<path>]

Initialize a new vault per §5.1. The vault root is <path> if supplied, otherwise the current directory. Exits non-zero if the target is not empty or already contains a head pinax.

8.2 mt add

mt add <fs-path>...

Ingest one or more files (or directories, recursively). For each file: deposit its bytes, mint an entry ID, emit core/blob-ref, core/path, core/fs-metadata, core/mime, and commit. Implementations MAY batch multiple files into a single state.

The recorded vault path for each file is the <fs-path> argument as given, lexically cleaned (redundant . components and repeated separators removed; no symlink or .. resolution against the filesystem) and NFC-normalized per §1.3; for directory ingest, the per-file path is the cleaned argument joined with the file’s path relative to it. Implementations MAY offer a --path <vault-path> override naming the recorded path explicitly (single-file ingest only).

Directory traversal policy: hidden files (dot-prefixed) are included; symbolic links to regular files are ingested by content (the target’s bytes); symbolic links to directories are not followed. Implementations MAY offer flags altering this policy; the defaults above are normative.

8.3 mt get

mt get <path|entry-id>

Write the bytes of the named entry’s current content blob to standard output. Resolution: argument matching the UUID v7 string form is an entry ID; otherwise it is treated as a path.

Note: the entry-ID interpretation wins outright — a vault path that is itself a valid UUID v7 string form is shadowed by this rule and unreachable through the string surface (the path interpretation is never tried, even when no entry has that ID). Implementations MAY reject creating such paths at ingest to close the footgun.

8.4 mt ls

mt ls [<path-prefix>]

List entries whose current paths begin with <path-prefix> (or all current paths if absent). The match is a plain octet-string prefix over NFC-normalized paths (§1.3); no path-component or separator semantics are implied.

8.5 mt mv

mt mv <from-path> <to-path>

Emit core/path facts retracting <from-path> and asserting <to-path> for the same entry, in a single commit. mv MUST fail without committing if <to-path> is currently bound to a different entry (rebinding at this surface is always explicit — see §4.2; a caller that intends to steal a path unlinks it first with rm).

mt link <existing-path> <new-path>

Emit a core/path fact asserting <new-path> for the entry currently at <existing-path>. link MUST fail without committing if <new-path> is currently bound to a different entry (see §8.5).

mt add (§8.2), by contrast, MAY bind an already-bound path: re-adding a file at the same vault path is the natural re-ingest workflow, and the superseded entry remains reachable by entry ID and through historical states.

8.7 mt rm

mt rm <path>

Emit a core/path fact retracting <path> from its current entry. The entry and its blob remain in the cella; the path is removed from the projection.

A --purge flag is reserved but not implemented in Phase 1 (no substrate delete; see §10).

8.8 mt facts

mt facts <path|entry-id>

List all facts attached to the entry, in chain order, with their hashes.

8.9 mt state

mt state [<stateref>]

Show metadata for a state: hash, created_at_ns, previous, fact count. Default argument: current.

8.10 mt log

mt log [--since <iso8601>] [--until <iso8601>] [-n <count>]

Walk the state chain from current head backward, emitting one line per state with its short hash, ISO-8601 timestamp, and fact count. Filters narrow the window.

8.11 mt as-of

mt as-of <stateref> <subcommand> [args...]

Run <subcommand> against a transient view of the vault projected to <stateref>. The durable head is not modified. Implementations MAY use a temporary index file or rebuild index.db and restore it after; either approach MUST NOT leave the persistent index in a state inconsistent with current head when the command exits.

Only read subcommands are meaningful here; implementations MUST reject writing verbs (add, mv, link, rm) under as-of.

8.12 mt reindex

mt reindex [--as-of <stateref>]

Rebuild index.db per §5.6. With --as-of, build against the named state instead of current head.

8.13 mt fsck

mt fsck

Verify: every blob reachable from current head is present in the cella; every state in the chain parses; every fact referenced by a state parses; the head pinax bytes encode a state present in the cella. Implementations MAY rely on syntheca and apotheca verification for byte-level integrity (no need to re-hash).

8.14 Reserved

The verbs gc and hook are reserved for future phases.

9 Errors

The error conditions defined by this specification are:

Implementations MAY surface additional implementation-defined errors (I/O failure, permission denied, …); such errors MUST be distinguishable from the protocol-defined errors above.

10 Out of scope (Phase 1)

The following are deliberately out of Phase 1 scope. Their absence is load-bearing for the v0.1 surface and consumers MUST NOT rely on them.

11 Future work (non-normative)