semopsis — Specification
version 0.1
This document specifies the semopsis protocol: a semantic-search
extension over a metatheca vault. semopsis
embeds the content blobs reachable from a metatheca state, organizes
the resulting vectors into an inverted-file (IVF) index whose
components are themselves deposita and pinakes on a dedicated
semopsis cella under the same vault root, and exposes
nearest-neighbour queries against any historical search-state.
The specification is language-agnostic. The reference implementation
is in Rust (semopsis-rs/).
semopsis is structurally an extension over metatheca, not a
parallel store. It introduces no new storage primitive — every byte
it writes is a syntheca depositum and every mutable pointer it holds
is an apotheca pinax — but it does occupy its own dedicated cella
under the metatheca vault root, at the fixed path
<vault>/ext/semopsis/cella/. What it adds is a view — a
particular organization of metatheca’s content optimized for semantic
queries. The name reflects this: ὄψις (opsis) means "view,
sight," and semopsis names the semantic view over the
underlying vault. The suffix mirrors English synopsis.
semopsis is the semantic half of a planned two-extension search
story. A sibling specification,
logopsis, defines a lexical
(inverted-index) extension with the same shape: its own dedicated
cella under the same vault root, an independent search-state chain,
an independent search-head pinax, and an independent advance cadence.
A coordinator layer above both — see
zetetes — performs hybrid retrieval by
querying both and fusing results. semopsis itself is a complete,
self-contained specification and MAY be deployed without logopsis
or zetetes.
For motivation, design choices, and the relationship to the
substrates and the sibling extension, see
README.md. For the substrate primitives
this document builds on, see
metatheca SPEC.md,
../../../syntheca/SPEC.md,
and
../../../apotheca/SPEC.md.
This document covers what sits above those: the index wire
formats, the embedding-model identity contract, the search-state
chain, the search-head pinax, the query algorithm, and the CLI
surface.
The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY in this document are to be interpreted as described in RFC 2119.
1 Scope and terminology
This document specifies semopsis Phase 1: a single local search index over a single metatheca vault, IVF-only (no product quantization, no graph indexes), with rebuild-only reindex semantics (no incremental posting-list mutation). Phase 1 is the v0.1 release surface; deferred items are listed in §10–§11.
Terminology not redefined here is inherited from metatheca SPEC §1, syntheca SPEC §1, and apotheca SPEC §1. In particular: cella, depositum, pinax, hash (BLAKE3), digest (SHA-256), vault, entry, fact, state, head.
1.1 Embedding
An embedding is a fixed-dimension vector of IEEE 754 binary32 floats, derived from a sequence of input bytes by a designated embedding model (§1.2). The dimension is fixed per search-state and recorded in the search-state blob (§4.3).
1.2 Embedding model and distance metric
An embedding model is identified by a tuple
(family, version, dimension) recorded in every search-state
blob (§4.3). The family is an opaque string
(e.g. nomic-embed-text); the version is an opaque string
(e.g. v1.5); the dimension is a positive integer.
A distance metric is identified by a string drawn from the
set {"cosine", "l2"}, recorded in every search-state blob
(§4.3). "cosine" denotes cosine
distance (1 − cos(a, b)); "l2" denotes Euclidean
distance. Under "cosine", if either operand has zero norm the
distance is defined as 1, so the metric is total over all
wire-legal embeddings.
Two embeddings are comparable iff they were produced by the same model identity and are being compared under the same distance metric. semopsis MUST refuse to compare embeddings across model identities, MUST refuse to query a search-state with a query vector produced by a different model, and MUST use the search-state’s recorded metric for all distance computations during indexing and querying against that search-state.
semopsis does not specify how embeddings are computed. The model is opaque to the protocol; the protocol guarantees only that the same model identity produces deterministic outputs for identical inputs.
A model MAY be partial: it MAY decline inputs it cannot meaningfully embed (binary content on a mixed vault being the canonical case). The decline decision is part of the model’s deterministic contract — identical input bytes under the same identity MUST always embed to the same vector or always be declined. Declined blobs are simply absent from the index (§5.2): they appear in no posting list and are never returned by a query.
1.3 Centroid
A centroid is the mean vector of a cluster of embeddings, produced by k-means clustering over a representative sample of the corpus’s embeddings (§5.2). Centroids share the dimension and model identity of the embeddings they summarize.
1.4 Posting list
A posting list is the set of (entry-ID, blob-hash, embedding) records assigned to a single centroid. Posting lists are immutable once written: they are content-addressed deposita (§4.2) and are replaced wholesale on reindex (§5.4).
1.5 Search-state
A search-state is a moment in the extension’s history. Each search-state is materialized as a content-addressed search-state blob (§4.3) capturing the centroid set, the posting-list set, the embedding-model identity, the distance metric, the metatheca state it was built against, and a back-pointer to the prior search-state. The set of search-states linked through these back-pointers is the search-state chain.
The genesis search-state is the unique search-state with
previous = null, created by smops init
(§7.1).
1.6 Search-head
The search-head is the single mutable pointer identifying the current search-state. It is realized as an apotheca pinax (§6) on the semopsis cella, holding the BLAKE3 hash of the current search-state blob.
The search-head pinax is disjoint from metatheca’s head pinax
(metatheca SPEC §6.1): the two pinakes live in disjoint cellae
(metatheca’s at <vault>/cella/, semopsis’s at
<vault>/ext/semopsis/cella/) and advance independently. This
independence is load-bearing: the semopsis search-state advances
when reindexing occurs (typically rare and expensive), while
metatheca’s head advances on every fact-emitting operation (frequent
and cheap). Where logopsis is also deployed, its search-head lives
in yet another cella (<vault>/ext/logopsis/cella/) and
advances independently of semopsis’s.
1.7 Search-state reference
A search-state reference (or search-stateref) follows the same grammar as metatheca’s stateref (metatheca SPEC §1.7), resolving against the search-state chain instead of the metatheca state chain:
a full BLAKE3 hash;
a hash prefix of at least 8 lowercase hex digits, unambiguous within the chain;
an ISO-8601 date or date-time, denoting the most recent search-state with
created_at_ns ≤ that_instant;a relative reference:
~or~NforNnon-negative;the literal
current.
A search-stateref MUST resolve to exactly one search-state or fail.
1.8 Index
semopsis uses index in this document to refer to the IVF index structure (centroids + posting lists) recorded in a search-state blob. The metatheca SQLite index (metatheca SPEC §1.8, §7) is unrelated and unaffected by semopsis. A logopsis inverted index, if deployed on the same vault, is also unrelated; its wire formats and search-states are governed by its own specification.
2 Relationship to metatheca and logopsis
semopsis is an extension over metatheca: it consumes the
content made addressable by the metatheca chain and adds a derived,
queryable view. It introduces no new storage primitive — every byte
is a syntheca depositum and every mutable pointer is an apotheca
pinax — but it does add a dedicated cella. All semopsis durable
state lives in a syntheca cella at the fixed vault-relative path
<vault>/ext/semopsis/cella/, separate from metatheca’s
source-of-truth cella at <vault>/cella/ (metatheca SPEC §7).
The split reflects the layers’ differing status: metatheca’s cella
is the source of truth and is never wholesale-deletable; semopsis’s
cella is wholly derived and MAY be discarded and rebuilt at any
time (§5.8).
The host/extension relationship has three concrete consequences:
Content authority lives in metatheca. semopsis MUST NOT mutate any metatheca-emitted fact, state blob, or the metatheca
headpinax. semopsis reads from metatheca’s cella to determine the set of content blobs to embed; it does not write back.semopsis writes are confined to semopsis’s cella. semopsis writes only deposita carrying semopsis’s wire formats (§4) and the single search-head pinax (§6) on its own cella. It MUST NOT write to any other cella in the vault.
The chains advance independently. A metatheca commit does not trigger a semopsis reindex, and a semopsis reindex does not require a metatheca commit.
smops reindexmay be invoked against any metatheca state in the chain; semopsis records which metatheca state it was built against (§4.3metatheca) so that any semopsis-state can be traced back to a precise point in the host’s history.
Where logopsis is also deployed, its state lives in yet another
dedicated cella at <vault>/ext/logopsis/cella/. semopsis
MUST NOT read from or write to it. Cross-extension
non-interference is enforced naturally by cella separation; the only
inter-extension contact point is the zetetes coordinator (zetetes
SPEC §2), which reads from both via their respective query
operations.
A vault MAY contain a metatheca chain without a semopsis chain. A
semopsis chain MUST NOT exist without a metatheca chain on the
same vault; smops init (§5.1) requires
metatheca’s head pinax to be present at <vault>/cella/.
A semopsis chain MAY exist with or without a logopsis chain on
the same vault.
3 Identifiers and time
3.1 Hashes
Content-addressing throughout semopsis uses syntheca’s BLAKE3 hash (syntheca SPEC §1.1, §1.2). Wire form: 64 lowercase hex digits.
3.2 Time
All timestamps follow metatheca SPEC §2.3: nanoseconds since the Unix epoch as a signed 64-bit integer, monotonically non-decreasing within a single vault process.
3.3 Embeddings on the wire
An embedding on the wire is the IEEE 754 binary32 little-endian
encoding of its dimension floats, concatenated with no
padding or framing. An embedding blob is exactly
4 * dimension octets. The endianness is fixed by this
specification; changing it is a protocol revision.
4 Wire formats
4.1 Canonical JSON
All JSON blobs defined by this specification (centroids manifest §4.4, posting list §4.2, search-state §4.3) are serialized as canonical JSON per RFC 8785, identical to metatheca SPEC §3.1. Where a blob carries binary payloads (centroid vectors, posting-list embeddings), those payloads are referenced by BLAKE3 hash, not embedded inline.
4.2 Posting-list blob
A posting-list blob is a canonical-JSON object:
{
"type": "semopsis/posting-list",
"version": 1,
"centroid": <integer>,
"model": { "family": "<utf8>", "version": "<utf8>", "dimension": <integer> },
"members": [
{
"entry": "<uuid-v7>",
"blob": "<blake3-hex>",
"embedding": "<blake3-hex>"
}
]
}
Field constraints:
typeMUST be the literal string"semopsis/posting-list".versionMUST be the integer1.centroidMUST be a non-negative integer matching the index of this posting list within the centroids manifest (§4.4).modelMUST equal themodelfield of the search-state blob referencing this posting list (§4.3).membersMUST be a JSON array of zero or more member records, sorted in strictly ascending order ofentry(UUID v7 string order, which equals byte order). The ordering is load-bearing: it gives one canonical byte form per assignment set, so identical rebuilds deduplicate on the cella (§5.3). Each member’sentryMUST be a UUID v7 string form (metatheca SPEC §2.1),blobMUST be the BLAKE3 hash of a depositum present in the cella, andembeddingMUST be the BLAKE3 hash of an embedding blob (§3.3) present in the cella.
A posting list’s identity is the BLAKE3 hash of its canonical-JSON bytes.
4.3 Search-state blob
A search-state blob is a canonical-JSON object:
{
"type": "semopsis/search-state",
"version": 1,
"previous": "<blake3-hex>" | null,
"metatheca": "<blake3-hex>",
"model": { "family": "<utf8>", "version": "<utf8>", "dimension": <integer> },
"metric": "cosine" | "l2",
"centroids": "<blake3-hex>",
"posting_lists": ["<blake3-hex>", "..."],
"created_at_ns": <integer>
}
Field constraints:
typeMUST be the literal string"semopsis/search-state".versionMUST be the integer1.previousMUST be either a BLAKE3 hash naming a search-state blob present in the cella, ornull(genesis search-state only). Exactly one search-state in any chain hasprevious = null.metathecaMUST be a BLAKE3 hash naming a metatheca state blob present in the cella (metatheca SPEC §3.3). It records which metatheca state this search-state was built against.modelMUST be the embedding model identity per §1.2.metricMUST be one of the literal strings"cosine"or"l2", identifying the distance metric per §1.2. The metric is fixed at genesis and inherited unchanged by every subsequent search-state in the chain (§5.4).centroidsMUST be the BLAKE3 hash of a centroids manifest (§4.4) present in the cella.posting_listsMUST be a JSON array of BLAKE3 hashes, each naming a posting-list blob (§4.2) present in the cella. The array length MUST equal the number of centroids declared in the centroids manifest. The array index MUST equal thecentroidfield of the referenced posting list.created_at_nsMUST be an integer per §3.2.
A search-state’s identity is the BLAKE3 hash of its canonical-JSON bytes.
4.4 Centroids manifest
A centroids manifest is a canonical-JSON object:
{
"type": "semopsis/centroids",
"version": 1,
"model": { "family": "<utf8>", "version": "<utf8>", "dimension": <integer> },
"vectors": ["<blake3-hex>", "..."]
}
Field constraints:
typeMUST be the literal string"semopsis/centroids".versionMUST be the integer1.modelMUST equal themodelfield of every search-state blob referencing this manifest.vectorsMUST be a JSON array of BLAKE3 hashes, each naming an embedding blob (§3.3) of dimensionmodel.dimensionpresent in the cella. The array index identifies the centroid; references from posting lists (§4.2centroid) and search-state blobs (§4.3posting_lists) are interpreted against this index.
A centroids manifest’s identity is the BLAKE3 hash of its canonical-JSON bytes.
The centroids manifest is separated from the search-state blob so that two search-states sharing the same centroids (e.g., an incremental rebuild that re-clustered to the same result, or a manifest reused under a different metatheca state) deduplicate naturally on the cella. Note that the manifest does not carry the distance metric: the same centroid vectors are valid under either metric, and the metric is recorded once per search-state.
5 Operations
A semopsis-equipped vault exposes the following protocol operations. Each is total: it terminates with one of the outcomes listed for it.
Throughout this section, deposit and fetch of semopsis blobs
(search-states, centroids manifests, posting lists, embedding blobs)
refer to operations on the semopsis cella at
<vault>/ext/semopsis/cella/. Operations that fetch metatheca
state blobs or content blobs target metatheca’s cella at
<vault>/cella/. Step-level cella targets are noted only where
the destination is non-obvious from context.
5.1 init
Creates a genesis search-state on a vault that already has a metatheca chain.
Verify metatheca’s
headpinax is present on metatheca’s cella at<vault>/cella/(§2); fail withNotFoundotherwise.Verify no semopsis cella exists at
<vault>/ext/semopsis/cella/; fail withConflictotherwise.Open a fresh syntheca cella at
<vault>/ext/semopsis/cella/, creating any missing parent directories. All subsequent deposits and the search-head pinax target this cella.Construct an empty centroids manifest (
vectors = []) for a caller-supplied model identity. Deposit it; letcbe the returned hash.Construct the genesis search-state blob:
previous = null,metatheca = <current metatheca head>,model = <as supplied>,metric = <as supplied>,centroids = c,posting_lists = [],created_at_ns = now(). Deposit it; letgbe the returned hash.Set the search-head pinax (§6) on the semopsis cella to
gwithexpected = None.
The genesis search-state contains no embeddings; it exists so that
reindex (§5.4) has a chain to advance from and
so that the model identity and distance metric are fixed at
init time.
On any failure prior to step 6, the implementation MUST leave no partially-initialized semopsis cella. If the cella was created in step 3, it MUST be removed before returning the failure.
5.2 Build embeddings
Given a metatheca state M and a model identity mid,
produce an embedding depositum for every content blob reachable from
M:
Walk
M’s cumulative fact set (metatheca SPEC §3.3) and collect the set ofcore/blob-refblobs current in the projection (metatheca SPEC §5.5).For each such blob hash
b: a. Compute the canonical embedding bytese = embed(get(b), mid)per the embedding-model contract (§1.2). If the model declinesb(§1.2), skip it: the blob contributes no embedding and appears in no posting list. b. Callsyntheca.deposit(e); record the returned BLAKE3 hash as the embedding hash forb.
Idempotent re-deposit (syntheca SPEC §2.1) is the expected case when
re-embedding a corpus that has not changed. Implementations
SHOULD realize step 2b via a HEAD-equivalent existence
check against the cella before transferring bytes, particularly for
remote backends; the syntheca contract permits this and the
bandwidth and Class-A-operation savings on object-storage backends
are substantial.
The embedding function embed is opaque to this specification.
Its only contract is determinism: identical input bytes under
identical mid MUST produce identical output bytes.
5.3 Cluster
Given an embedding set produced by §5.2 and
a distance metric mt, partition it into K clusters by
k-means and emit a centroids manifest:
Choose
K. Implementations SHOULD useK ≈ √NwhereNis the number of embeddings, clamped to[1, N]forN ≥ 1. WhenN = 0(the empty corpus),K = 0: the centroids manifest is empty and the search-state carries no posting lists, mirroring the genesis shape (§5.1).Run k-means under metric
mtto convergence or to an implementation-defined iteration cap. Clustering MUST be deterministic within an implementation: the seed MUST be derived from the metatheca state hash being indexed, so the same implementation (at the same version) reindexing the same metatheca state under the same model and metric produces a byte-identical centroids manifest, and adjacent identical reindexes deduplicate on the cella. Byte-identical clustering across implementations is NOT required — floating-point summation order and iteration caps legitimately differ — and a chain’s identity is its model and metric, never its clustering.For each centroid vector, encode it per §3.3 and deposit it; let
v_ibe the returned hash for centroidi.Construct the centroids manifest with
vectors = [v_0, …, v_{K-1}]and the supplied model identity. Deposit it; return the manifest hash.
5.4 Reindex
Atomically advances search-head from the current search-state to a new search-state built against a chosen metatheca state.
Inputs: a metatheca stateref (default: metatheca’s current head).
Resolve the metatheca stateref per metatheca SPEC §5.3 to a metatheca state hash
M.Read the current search-state blob via the search-head pinax (§6); extract its model identity
midand distance metricmt. Compute the storage digestdof the search-head pinax bytes:sha256over the bytes returned bysyntheca.get_pinax, or remembered from the prior successfulset_pinax(apotheca SPEC §2.5/§3.5expectedsemantics).Build embeddings per §5.2 against
Mundermid.Cluster per §5.3 under
mt; obtain the centroids manifest hashcand the countK.Assign each embedding to its nearest centroid under metric
mt, ties broken by the lowest centroid index. For each centroidi ∈ [0, K), construct a posting-list blob (§4.2) withcentroid = i,model = mid, andmembersenumerating the embeddings assigned to it, sorted ascending byentry(§4.2). Deposit each posting-list blob; letp_ibe its hash.Construct the new search-state blob:
previous = <current search-head>,metatheca = M,model = mid,metric = mt,centroids = c,posting_lists = [p_0, …, p_{K-1}],created_at_ns = now(). Deposit it; letsbe the returned hash.Call
syntheca.set_pinax(name = "head", bytes = s_bytes, expected = Some(d))against the semopsis cella.
The commit point is step 7. Steps 1–6 are pre-commit and produce only content-addressed deposita that are safe under interruption.
On Conflict from step 7, another writer advanced search-head
concurrently. The caller MAY retry from step 2.
The model identity and distance metric are fixed by the
genesis search-state and inherited unchanged by every subsequent
search-state. reindex MUST refuse to write a search-state
with a model or metric field that differs from the
current search-state’s. Changing either is out of Phase 1 scope
(§10).
5.5 Query
Given a query byte sequence and a search-stateref, return the top
k nearest entries.
Resolve the search-stateref per §1.7 to a search-state hash
s. Fetch and parse the search-state blob; extract its model identitymidand distance metricmt.Embed the query under
mid. Implementations MUST verify that the embedding-model used for the query matchesmid; mismatch is aModelMismatcherror.Fetch and parse the centroids manifest named by the search-state. For each centroid, fetch its embedding blob and compute distance to the query embedding under
mt.Select the
nprobenearest centroids, ties broken by the lowest centroid index (nprobeis a query parameter; default is implementation-defined within[1, K]).For each selected centroid
i, fetch the posting-list blobposting_lists[i]and, for each member, fetch the member’s embedding blob and compute distance to the query embedding undermt.Return the top
kmembers by ascending distance, ties broken by ascendingentry, each as(entry, blob, distance). The ordering is total, so identical queries against the same search-state with the samenprobereturn identical results.
Query is read-only: it MUST NOT modify any depositum or
pinax.
5.6 Resolve a search-state reference
Given a search-stateref (§1.7) and the current search-head, return the unique search-state hash it denotes. Algorithmically identical to metatheca SPEC §5.3 with the search-state chain substituted for the metatheca state chain.
5.7 Walk the search-state chain
Algorithmically identical to metatheca SPEC §5.4 with the search-state chain substituted.
5.8 Destroy
Removes the semopsis cella and all semopsis state from the vault.
Inputs: none.
If
<vault>/ext/semopsis/cella/is absent, fail withNotFound.Recursively remove the directory
<vault>/ext/semopsis/(including the cella and any extension-private auxiliary state under it).
Destroy is total: on completion, no semopsis state remains on the
vault. metatheca state and any sibling extension state are
unaffected. A subsequent smops init (§5.1)
succeeds if and only if destroy completed.
Destroy is the supported mechanism for switching model identity or
distance metric (§10): destroy the chain, then
re-run smops init with the desired identity. It is also the
recovery mechanism for a corrupted semopsis cella, since the chain
is wholly derivable from metatheca’s cella by reindexing — though
re-embedding the corpus is the dominant cost and a known consequence
of destroy.
Destroy MUST NOT touch <vault>/cella/,
<vault>/index.db, or any other entry in <vault>/ext/.
5.9 Auxiliary state
An implementation MAY maintain extension-private
auxiliary state — caches, projections, or other derived
structures that accelerate its own operation — under
<vault>/ext/semopsis/, outside the cella.
§5.8 already provides for the removal of any such
state; this section states the contract that makes it legitimate.
Observational neutrality. Auxiliary state MUST NOT change the observable behaviour of any protocol operation: a query answered from auxiliary state MUST return results identical to the same query evaluated from the wire-format blobs (§4) — the same hits, the same order, the same scores. In particular, substituting a different embedding function or distance computation inside an acceleration structure violates the model-identity and metric contract (§1.2), however convenient the substitute.
Disposability. Auxiliary state MUST be wholly derived: deleting it at any moment MUST NOT affect correctness, only performance. It is rebuilt or repaired only by the implementation that owns it.
Containment. Auxiliary state MUST NOT be written into any cella, referenced from any wire-format blob, or otherwise become interchange surface. Another implementation MUST be able to operate on the same vault while ignoring it entirely.
Freshness. An implementation SHOULD key auxiliary state to the hash of the search-state it reflects: content addressing makes staleness detection exact, and anything the auxiliary state cannot serve — historical search-states, a stale or damaged structure — falls back to the wire-format blobs.
Maintaining no auxiliary state is always conformant (§9); these constraints bind only implementations that choose acceleration.
The reference implementation maintains a SQLite projection at
<vault>/ext/semopsis/index.db, written by reindex from its in-memory index and keyed by the new search-state’s hash. It packs all centroid vectors into one blob and each cluster’s member vectors into another, replacing the per-vector blob fan-out that dominates query cost on the wire-format path; queries against any other search-state, or with a stale or corrupt projection, silently take the wire-format path. Equivalence between the two paths is enforced by a test sweep over query shapes, result sizes, evaluation strategies, and historical states — the recommended conformance technique for any implementation that adds acceleration. Deployment analysis and measurements:SCALE.mdin this directory.
6 Search-head pinax
6.1 Name
The search-head pinax MUST be stored on the semopsis cella under
the apotheca pinax name head (four ASCII octets: h,
e, a, d). This name is fixed by this
specification.
The name is a valid apotheca name (apotheca SPEC §4.1): non-empty,
length 4, no /, no NUL. semopsis therefore imposes no
multi-segment-name requirement on the underlying apotheca
implementation.
The pinax shares only its name with metatheca’s head
(metatheca SPEC §6.1) and with any sibling extension’s search-head;
the pinakes live in disjoint cellae and never collide. The same
naming convention is used by logopsis (head on the logopsis
cella) and zetetes (head on the zetetes cella).
6.2 Bytes
The search-head pinax bytes follow metatheca SPEC §6.2 verbatim: the 64-octet ASCII encoding of the current search-state’s BLAKE3 hash as 64 lowercase hex digits, with no trailing newline or whitespace.
6.3 Compare-and-swap protocol
All updates to search-head MUST go through
syntheca.set_pinax with expected set to the SHA-256
digest of the current search-head
bytes, identical in mechanism to metatheca SPEC §6.3. The
genesis case (init, §5.1) sets search-head with
expected = None.
7 CLI surface
The reference CLI binary is semopsis with the recommended
alias smops. It exposes the protocol against the vault rooted
at the current working directory unless overridden with
--vault <PATH>.
This section specifies the protocol-bearing subcommands. Diagnostic output formatting is implementation-defined unless otherwise noted.
7.1 smops init
smops init --model <family>:<version>:<dimension> --metric <cosine|l2> [<path>]
Initialize a semopsis chain on a vault per §5.1. The
vault root is <path> if supplied, otherwise the current
directory. The vault MUST already contain a metatheca chain.
Exits non-zero if metatheca is absent or a semopsis chain already
exists.
--metric defaults to cosine if omitted;
implementations SHOULD warn that the default has been used so the
caller is aware the metric is now fixed for the lifetime of the
chain.
7.2 smops reindex
smops reindex [--as-of <metatheca-stateref>]
Build a new search-state per §5.4 against the named metatheca state (default: metatheca’s current head). On success, prints the new search-state hash to standard output.
7.3 smops query
smops query [--k <int>] [--nprobe <int>] [--as-of <search-stateref>] <query>
smops query [--k <int>] [--nprobe <int>] [--as-of <search-stateref>] -
Run a query per §5.5. <query> is the query text;
- reads query bytes from standard input. Default --k
is 10; default --nprobe is implementation-defined. Default
--as-of is current.
Output format per result, one line:
<entry-uuid> <blob-hash> <distance>
7.4 smops state
smops state [<search-stateref>]
Show metadata for a search-state: hash, created_at_ns,
previous, metatheca, model identity, distance metric,
centroid count, total member count. Default argument:
current.
7.5 smops log
smops log [--since <iso8601>] [--until <iso8601>] [-n <count>]
Walk the search-state chain from current search-head backward, emitting one line per search-state with its short hash, ISO-8601 timestamp, model identity, distance metric, and metatheca state short hash.
7.6 smops fsck
smops fsck
Verify: the search-head pinax bytes encode a search-state present in
the cella; every search-state in the chain parses; the model
identity and metric are constant across the chain; every centroids
manifest parses and references embedding blobs of the declared
dimension; every search-state’s posting_lists array length
equals its manifest’s centroid count, and each referenced posting
list’s centroid field equals its array index; every posting
list parses, its members are sorted strictly ascending by
entry, its members’ embedding blobs are present and of the
declared dimension, and its members’ content blobs (blob
field) are present; every search-state’s metatheca field
names a metatheca state present in the cella. Implementations
MAY rely on syntheca and apotheca verification for byte-level
integrity.
7.7 smops destroy
smops destroy [--force] [<path>]
Destroy the semopsis cella per §5.8. The vault root
is <path> if supplied, otherwise the current directory.
Unless --force is supplied, implementations SHOULD prompt
for confirmation before proceeding, since destroy is irreversible
without a metatheca-side reindex (§5.4) to rebuild
the chain — and re-embedding is the dominant cost. Exits non-zero if
no semopsis cella exists at the resolved path.
7.8 Reserved
The verb gc is reserved for future phases.
8 Errors
The error conditions defined by this specification are:
NotFound— a search-stateref or referenced blob did not resolve.Ambiguous— a hash-prefix search-stateref matched more than one search-state.Conflict— search-head CAS lost a race; the caller MAY retry (§5.4).Malformed— a wire-format blob did not parse per §4, or an argument did not parse per §1.7.ModelMismatch— a query was issued against a search-state whose model identity differs from the model used to embed the query (§5.5), or a reindex attempted to change the model or metric fixed at genesis (§5.4).IntegrityError— inherited from syntheca / apotheca; surfaces unchanged.
Implementations MAY surface additional implementation-defined errors (I/O failure, embedding-model unavailable, …); such errors MUST be distinguishable from the protocol-defined errors above.
9 Conformance
A Phase 1 implementation MUST implement:
§2 (host/extension relationship)
§3 (identifiers, time, embedding wire format)
§5.1–§5.8 (operations), including metric pinning and inheritance, and the destroy operation
§5.9 (auxiliary state), applicable only when an implementation maintains such state
§6 (search-head pinax)
§7 (CLI), with the exception that
--as-ofMAY be omitted fromsmops queryif the implementation always queriescurrent§8 (errors)
A Phase 1 implementation MAY omit:
§10 items.
10 Out of scope (Phase 1)
The following are deliberately out of Phase 1 scope. Their absence is load-bearing for the v0.1 surface and consumers MUST NOT rely on them.
Model and metric migration. A search-state’s model identity and distance metric are immutable across the chain. Switching either requires destroying the semopsis chain (§5.8) and re-running
smops init. A future phase may define rotation operations for one or both that preserve the chain.Incremental reindex. Phase 1 reindex rebuilds all posting lists from scratch. Content-addressing means unchanged posting lists deduplicate at the cella, but the work of recomputing assignments is full each time.
Embedding packs. Phase 1 stores each embedding as its own content-addressed blob, referenced by hash from posting lists. This produces a per-member fan-out at query time that is fast on local cellae but adds latency on object-storage cellae (one GET per examined embedding). A future phase may define a packed-embeddings depositum format that stores all of a posting list’s embeddings in a single blob; this is purely additive and does not change the search-state wire format. See §11.
Product quantization, OPQ, HNSW, or other ANN structures. Phase 1 is IVF-flat only.
Reranking with the original content. Phase 1 returns candidates ranked by embedding distance. Content-aware reranking is a caller concern.
Garbage collection. Old search-states and their referenced posting lists, centroids, and embeddings remain on the cella indefinitely. Blocked by absence of delete in apotheca and syntheca.
Cross-vault index sharing. A search-state references a metatheca state hash; in principle, two vaults sharing the same metatheca chain prefix could share search-states. This is not specified.
Hybrid keyword+vector ranking. Single query vector, single result list. Hybrid retrieval combining semopsis’s semantic results with logopsis’s lexical results is the responsibility of the zetetes coordinator layer; it is not part of the semopsis protocol.
Query expansion, multi-vector queries. Single query vector per call.
11 Future work (non-normative)
Embedding packs (alluded to in §10). A
semopsis/embedding-packdepositum kind whose payload isKconcatenated embeddings in a fixed order, referenced by a packed-posting-list variant alongside per-member entry/blob metadata. Reduces a query’s remote GET count fromO(N/K · nprobe)toO(nprobe), which is the difference between viable and unviable IVF-on-object-storage.A
core/embedding-reffact in metatheca, recording embedding hashes alongsidecore/blob-ref. This would let semopsis reuse embeddings across reindexes without recomputation when the underlying blob is unchanged, collapsing the steady-state reindex of an unchanged corpus to a pure reclustering pass. Additive change to metatheca.Skip-list pointers in the search-state blob, mirroring metatheca SPEC §11, to make as-of resolution sublinear.
nproberecorded in the search-state blob. Phase 1’s defaultnprobeis implementation-defined (§5.5 step 4), so the same query against the same search-state can yield different results across implementations. A future-phase addition would record annprobe_defaultfield in the search-state, making query reproducibility a property of the protocol rather than the implementation. Cf. zetetes SPEC §11 onk_internal.Phase 2 model rotation: a new search-state kind that records both an old and a new model identity, allowing graceful migration without destroying the chain.
Phase 2 metric rotation: as above for the distance metric. Less commonly needed in practice than model rotation but specifiable with the same machinery.
Phase 2 incremental reindex: posting-list-level deltas keyed by centroid identity stable across rebuilds.
A remote-cella backend (apotheca §10) makes the IVF posting-list-per-file layout natural for object-storage deployment; combined with embedding packs, no further semopsis-spec change is required for production-suitable R2 deployment.
A remote query service. Even with embedding packs, remote IVF-flat still moves
nprobe/Kof the vector bytes per query; a companion serving tier (hyperetes, the servant) that embeds and evaluates §5.5 beside the cella and returns only the top-kmakes the client cost independent of corpus size and dimension. It owns no chain state and is not an extension — pure serving infrastructure over the query surface. The constraint it feeds back into this spec: Phase 2 query cores should stay I/O-agnostic (byte-range reads behind an async seam) and compilable to WASM, so a conformant evaluator — including the embedding step — can run in an edge runtime at zero distance from the cella. Deployment analysis and measurements:SCALE.mdin this directory, scenario 4.