logopsis — Specification
version 0.1
This document specifies the logopsis protocol: a lexical-search
extension over a metatheca vault. logopsis
tokenizes the content blobs reachable from a metatheca state,
organizes the resulting terms into an inverted index whose
components are themselves deposita and pinakes on a dedicated
logopsis cella under the same vault root, and exposes BM25-ranked
queries against any historical search-state.
The specification is language-agnostic. The reference implementation
is in Rust (logopsis-rs/).
logopsis is structurally an extension over metatheca, not a
parallel store. It introduces no new storage primitive — every byte
it writes is a syntheca depositum and every mutable pointer it
holds is an apotheca pinax — but it does occupy its own dedicated
cella under the metatheca vault root, at the fixed path
<vault>/ext/logopsis/cella/. What it adds is a view — a
particular organization of metatheca’s content optimized for
lexical queries. The name reflects this: ὄψις (opsis) means
"view, sight," and logopsis names the lexical view (from λόγος,
"word") over the underlying vault. The suffix mirrors English
synopsis.
logopsis is the lexical half of a planned two-extension search
story. A sibling specification, semopsis,
defines a semantic (vector-embedding) extension with the same shape:
its own dedicated cella under the same vault root, an independent
search-state chain, an independent search-head pinax, and an
independent advance cadence. A coordinator
layer above both — see zetetes — performs
hybrid retrieval by querying both and fusing results. logopsis itself
is a complete, self-contained specification and MAY be deployed
without semopsis or zetetes.
For motivation, design choices, and the relationship to the
substrates and the sibling extension, see
README.md.
For the substrate primitives this document builds on, see
metatheca SPEC.md,
../../../syntheca/SPEC.md,
and
../../../apotheca/SPEC.md.
This document covers
what sits above those: the index wire formats, the analyzer
identity contract, the search-state chain, the search-head pinax,
the query algorithm, and the CLI surface.
The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY in this document are to be interpreted as described in RFC 2119.
1 Scope and terminology
This document specifies logopsis Phase 1: a single local
search index over a single metatheca vault, BM25-ranked, with
rebuild-only reindex semantics (no incremental posting-list
mutation), a single fixed analyzer identity per chain, and
conjunctive (AND) / disjunctive (OR) / phrase /
negation query operators only. Phase 1 is the v0.1 release surface;
deferred items are listed in
§10–§11.
Terminology not redefined here is inherited from metatheca SPEC §1, syntheca SPEC §1, and apotheca SPEC §1. In particular: cella, depositum, pinax, hash (BLAKE3), digest (SHA-256), vault, entry, fact, state, head.
1.1 Token
A token is a finite sequence of Unicode scalar values produced by applying the chain’s analyzer (§1.2) to a content blob. Tokens are the unit of indexing: posting lists key on tokens, queries match on tokens.
1.2 Analyzer
An analyzer is identified by a tuple
(family, version, language) recorded in every search-state
blob (§4.4). The family is an opaque string
(e.g. unicode-segmentation); the version is an opaque string
(e.g. v1.12); the language is a BCP 47 language tag (e.g.
en, sr-Latn, mul for multilingual) or the
literal string und for analyzer chains that do not perform
language-specific normalization.
The analyzer’s contract is to be a deterministic pure function from input bytes to a sequence of tokens with associated positions. Two invocations of the same analyzer identity on identical bytes MUST produce byte-identical token sequences and position sequences.
logopsis MUST refuse to query a search-state with tokens produced
by a different analyzer identity, surfaced as AnalyzerMismatch
(§8). Analyzer identity is fixed by the
genesis search-state and inherited unchanged by every subsequent
search-state in the chain. Changing analyzers is out of Phase 1
scope (§10).
logopsis does not specify what an analyzer does internally (tokenization rules, case folding, stemming, stopword removal, Unicode normalization, n-gram generation). These are implementation-defined within the analyzer family/version. The protocol guarantees only that token sequences are reproducible under a recorded analyzer identity.
1.3 Term
A term is a token as it appears in the term dictionary (§4.5). In Phase 1, term and token are the same string; the distinction exists so future phases may introduce token-to-term mappings (e.g. synonym folding) without renaming the dictionary structure.
1.4 Posting list
A posting list for a term t is the ordered set of
(entry-ID, blob-hash, term-frequency, positions) records for every
blob whose analyzer output contains t. Posting lists are
immutable once written: they are content-addressed deposita
(§4.3) and are replaced wholesale on reindex
(§5.4).
1.5 Term dictionary
The term dictionary is the sorted association from terms to posting-list hashes for a given search-state. It is materialized as a single content-addressed depositum (§4.5).
1.6 Document statistics
A document statistics blob records, for a given search-state, the corpus-level quantities BM25 requires: total document count, total token count, average document length in tokens, and the per-document length of every indexed blob. It is materialized as a single content-addressed depositum (§4.6).
1.7 Search-state
A search-state is a moment in the extension’s history. Each search-state is materialized as a content-addressed search-state blob (§4.4) capturing the term dictionary, the document statistics, the analyzer identity, the BM25 parameter set, the metatheca state it was built against, and a back-pointer to the prior search-state. The set of search-states linked through these back-pointers is the search-state chain.
The genesis search-state is the unique search-state with
previous = null, created by lgops init
(§7.1).
1.8 Search-head
The search-head is the single mutable pointer identifying the current search-state. It is realized as an apotheca pinax (§6) on the logopsis cella, holding the BLAKE3 hash of the current search-state blob.
The search-head pinax is disjoint from metatheca’s head pinax
(metatheca SPEC §6.1): the two pinakes live in disjoint cellae
(metatheca’s at <vault>/cella/, logopsis’s at
<vault>/ext/logopsis/cella/) and advance independently. This
independence is load-bearing: the logopsis search-state advances
when reindexing occurs, while metatheca’s head advances on every
fact-emitting operation. logopsis reindex is comparatively cheap
(tokenization is fast) and so the chain MAY advance more
frequently than semopsis’s, but the cadence is left to deployment
policy. Where semopsis is also deployed, its search-head lives in
yet another cella (<vault>/ext/semopsis/cella/) and advances
independently of logopsis’s.
1.9 Search-state reference
A search-state reference (or search-stateref) follows the same grammar as metatheca’s stateref (metatheca SPEC §1.7), resolving against the search-state chain instead of the metatheca state chain:
a full BLAKE3 hash;
a hash prefix of at least 8 lowercase hex digits, unambiguous within the chain;
an ISO-8601 date or date-time, denoting the most recent search-state with
created_at_ns ≤ that_instant;a relative reference:
~or~NforNnon-negative;the literal
current.
A search-stateref MUST resolve to exactly one search-state or fail.
1.10 BM25 parameters
The BM25 parameters are the two tunable scalars k1
and b of the Okapi BM25 ranking function (Robertson & Walker
1994, Robertson & Zaragoza 2009). Phase 1 logopsis uses the
standard BM25 formula:
score(d, q) = Σ_{t ∈ q} idf(t) · (tf(t, d) · (k1 + 1))
/ (tf(t, d) + k1 · (1 − b + b · |d| / avgdl))
with idf(t) = ln((N − df(t) + 0.5) / (df(t) + 0.5) + 1), where
N is the total document count, df(t) is the number of
documents containing t, tf(t, d) is the term frequency
of t in d, |d| is the length of d in
tokens, and avgdl is the average document length.
k1 and b are recorded in every search-state blob
(§4.4) and fixed for the lifetime of the
chain like the analyzer identity. Default values, applied when
lgops init (§7.1) is invoked without explicit
parameters: k1 = 1.2, b = 0.75.
1.11 Index
logopsis uses index in this document to refer to the inverted index structure (term dictionary + posting lists + document statistics) recorded in a search-state blob. The metatheca SQLite index (metatheca SPEC §1.8, §7) is unrelated and unaffected by logopsis. A semopsis IVF index, if deployed on the same vault, is also unrelated; its wire formats and search-states are governed by its own specification.
2 Relationship to metatheca and semopsis
logopsis is an extension over metatheca: it consumes the
content made addressable by the metatheca chain and adds a derived,
queryable view. It introduces no new storage primitive — every byte
is a syntheca depositum and every mutable pointer is an apotheca
pinax — but it does add a dedicated cella. All logopsis durable
state lives in a syntheca cella at the fixed vault-relative path
<vault>/ext/logopsis/cella/, separate from metatheca’s
source-of-truth cella at <vault>/cella/ (metatheca SPEC §7).
The split reflects the layers’ differing status: metatheca’s cella
is the source of truth and is never wholesale-deletable; logopsis’s
cella is wholly derived and MAY be discarded and rebuilt at any
time (§5.8).
The host/extension relationship has three concrete consequences:
Content authority lives in metatheca. logopsis MUST NOT mutate any metatheca-emitted fact, state blob, or the metatheca
headpinax. logopsis reads from metatheca’s cella to determine the set of content blobs to tokenize; it does not write back.logopsis writes are confined to logopsis’s cella. logopsis writes only deposita carrying logopsis’s wire formats (§4) and the single search-head pinax (§6) on its own cella. It MUST NOT write to any other cella in the vault.
The chains advance independently. A metatheca commit does not trigger a logopsis reindex, and a logopsis reindex does not require a metatheca commit.
lgops reindexmay be invoked against any metatheca state in the chain; logopsis records which metatheca state it was built against (§4.4metatheca) so that any logopsis-state can be traced back to a precise point in the host’s history.
Where semopsis is also deployed, its state lives in yet another
dedicated cella at <vault>/ext/semopsis/cella/. logopsis
MUST NOT read from or write to it. Cross-extension
non-interference is enforced naturally by cella separation; the only
inter-extension contact point is the zetetes coordinator (zetetes
SPEC §2), which reads from both via their respective query
operations.
A vault MAY contain a metatheca chain without a logopsis chain. A
logopsis chain MUST NOT exist without a metatheca chain on the
same vault; lgops init (§5.1) requires metatheca’s
head pinax to be present at <vault>/cella/. A logopsis
chain MAY exist with or without a semopsis chain on the same
vault.
3 Identifiers and time
3.1 Hashes
Content-addressing throughout logopsis uses syntheca’s BLAKE3 hash (syntheca SPEC §1.1, §1.2). Wire form: 64 lowercase hex digits.
3.2 Time
All timestamps follow metatheca SPEC §2.3: nanoseconds since the Unix epoch as a signed 64-bit integer, monotonically non-decreasing within a single vault process.
3.3 Strings
All term strings on the wire are valid UTF-8. Implementations MUST
reject any analyzer output that is not valid UTF-8 with
Malformed. Term-internal byte ordering is the natural
lexicographic ordering of UTF-8 bytes; this ordering is used
wherever sorting is required (§4.5).
4 Wire formats
4.1 Canonical JSON
All JSON blobs defined by this specification (search-state §4.4, term dictionary §4.5, document statistics §4.6) are serialized as canonical JSON per RFC 8785, identical to metatheca SPEC §3.1. Where a blob carries a tabular payload (the posting list of §4.3), that payload is encoded in a binary format defined here and referenced by BLAKE3 hash, not embedded as JSON.
4.2 Integer encoding for binary payloads
Where this specification requires variable-length integer encoding in a binary payload, it uses LEB128 unsigned (Itanium ABI / DWARF ULEB128) for non-negative integers. Fixed-width integers are little-endian.
4.3 Posting-list blob (binary)
A posting-list blob is the binary encoding of an ordered sequence of (entry-ID, blob-hash, term-frequency, positions) records. The blob header is:
+--------+--------+--------+--------+
| magic: 4 octets "LXPL" |
+--------+--------+--------+--------+
| version: u32 LE = 1 |
+--------+--------+--------+--------+
| record_count: ULEB128 |
+-----------------------------------+
| records: <record_count> records |
+-----------------------------------+
Each record is:
+-----------------------------------+
| entry: 16 octets, UUID v7 binary |
+-----------------------------------+
| blob: 32 octets, BLAKE3 raw |
+-----------------------------------+
| tf: ULEB128 |
+-----------------------------------+
| pos_count: ULEB128 |
+-----------------------------------+
| positions: <pos_count> ULEB128 |
+-----------------------------------+
Field constraints:
magicMUST be the four ASCII octetsLXPL(logopsis posting list).versionMUST be the little-endian u321.record_countMUST equal the number of records that follow.Records MUST appear in ascending order of
entry(UUID v7 sort order, equivalent to ascending generation time).entryis the binary form of a UUID v7 (16 octets).blobis the raw 32-octet BLAKE3 hash of a depositum present in the cella.tfis the term frequency: the count of positions, equal topos_count. It is encoded redundantly so that scoring can proceed without scanning the position array.pos_countMUST equaltf.positionsis the ordered sequence of zero-based token offsets in the document at which the term occurs. Positions MUST appear in strictly ascending order. Positions are absolute, not delta-encoded; delta encoding is deferred to a future phase.
A posting list’s identity is the BLAKE3 hash of these bytes.
The binary format is chosen over JSON for posting lists because posting lists dominate the index byte size, are read in tight loops during query evaluation, and have a fixed-shape record structure that JSON would inflate by 5–10x with no benefit.
4.4 Search-state blob
A search-state blob is a canonical-JSON object:
{
"type": "logopsis/search-state",
"version": 1,
"previous": "<blake3-hex>" | null,
"metatheca": "<blake3-hex>",
"analyzer": { "family": "<utf8>", "version": "<utf8>", "language": "<bcp47>" },
"bm25": { "k1": <number>, "b": <number> },
"dictionary": "<blake3-hex>",
"docstats": "<blake3-hex>",
"created_at_ns": <integer>
}
Field constraints:
typeMUST be the literal string"logopsis/search-state".versionMUST be the integer1.previousMUST be either a BLAKE3 hash naming a search-state blob present in the cella, ornull(genesis search-state only). Exactly one search-state in any chain hasprevious = null.metathecaMUST be a BLAKE3 hash naming a metatheca state blob present in the cella (metatheca SPEC §3.3).analyzerMUST be the analyzer identity per §1.2.bm25MUST containk1andbas JSON numbers per §1.10. Both MUST be non-negative finite IEEE 754 binary64 values;bMUST additionally satisfy0 ≤ b ≤ 1. The pair is fixed at genesis and inherited unchanged by every subsequent search-state.dictionaryMUST be the BLAKE3 hash of a term-dictionary blob (§4.5) present in the cella.docstatsMUST be the BLAKE3 hash of a document-statistics blob (§4.6) present in the cella.created_at_nsMUST be an integer per §3.2.
A search-state’s identity is the BLAKE3 hash of its canonical-JSON bytes.
4.5 Term-dictionary blob
A term-dictionary blob is a canonical-JSON object:
{
"type": "logopsis/dictionary",
"version": 1,
"analyzer": { "family": "<utf8>", "version": "<utf8>", "language": "<bcp47>" },
"terms": [
{ "term": "<utf8>", "df": <integer>, "posting": "<blake3-hex>" }
]
}
Field constraints:
typeMUST be the literal string"logopsis/dictionary".versionMUST be the integer1.analyzerMUST equal theanalyzerfield of every search-state blob referencing this dictionary.termsMUST be a JSON array sorted in strictly ascending UTF-8 byte order of thetermfield. Each entry’sdfMUST be a positive integer equal to the number of records in the referenced posting list. Each entry’spostingMUST be the BLAKE3 hash of a posting-list blob (§4.3) present in the cella.
A term-dictionary’s identity is the BLAKE3 hash of its canonical-JSON bytes.
The dictionary is separated from the search-state blob so that two search-states sharing the same dictionary deduplicate naturally on the cella. In practice, dictionaries differ across reindexes whenever any document is added, removed, or modified, so this deduplication is rarer than the centroid-manifest deduplication of semopsis; it is preserved nonetheless for symmetry and for the edge case of reindexing the same metatheca state under the same analyzer.
4.6 Document-statistics blob
A document-statistics blob is a canonical-JSON object:
{
"type": "logopsis/docstats",
"version": 1,
"analyzer": { "family": "<utf8>", "version": "<utf8>", "language": "<bcp47>" },
"doc_count": <integer>,
"total_tokens": <integer>,
"avgdl": <number>,
"lengths": [
{ "entry": "<uuid-v7>", "blob": "<blake3-hex>", "len": <integer> }
]
}
Field constraints:
typeMUST be the literal string"logopsis/docstats".versionMUST be the integer1.analyzerMUST equal theanalyzerfield of every search-state blob referencing this docstats.doc_countMUST equal the length oflengths.total_tokensMUST equal the sum of alllenvalues inlengths.avgdlMUST equaltotal_tokens / doc_countas IEEE 754 binary64, except that whendoc_count = 0(a docstats blob over an empty corpus, including the genesis docstats of §5.1)avgdlMUST be0. Implementations MUST recompute and verify on read.lengthsMUST be a JSON array sorted in ascending order ofentry(UUID v7 sort order). Each entry’sblobis the BLAKE3 hash of the content blob whose length in tokens (under the recorded analyzer) islen.
A document-statistics blob’s identity is the BLAKE3 hash of its canonical-JSON bytes.
The docstats blob is separated from the search-state blob so that adjacent reindexes of an unchanged corpus deduplicate, and so that the search-state blob itself remains small (just hashes, like semopsis’s).
5 Operations
A logopsis-equipped vault exposes the following protocol operations. Each is total: it terminates with one of the outcomes listed for it.
Throughout this section, deposit and fetch of logopsis blobs
(search-states, dictionaries, posting lists, docstats) refer to
operations on the logopsis cella at
<vault>/ext/logopsis/cella/. Operations that fetch metatheca
state blobs or content blobs target metatheca’s cella at
<vault>/cella/. Step-level cella targets are noted only where
the destination is non-obvious from context.
5.1 init
Creates a genesis search-state on a vault that already has a metatheca chain.
Verify metatheca’s
headpinax is present on metatheca’s cella at<vault>/cella/(§2); fail withNotFoundotherwise.Verify no logopsis cella exists at
<vault>/ext/logopsis/cella/; fail withConflictotherwise.Open a fresh syntheca cella at
<vault>/ext/logopsis/cella/, creating any missing parent directories. All subsequent deposits and the search-head pinax target this cella.Construct an empty term-dictionary blob (
terms = []) for a caller-supplied analyzer identity. Deposit it; letdictbe the returned hash.Construct an empty document-statistics blob (
doc_count = 0,total_tokens = 0,avgdl = 0,lengths = []) for the same analyzer identity. Deposit it; letdsbe the returned hash.Construct the genesis search-state blob:
previous = null,metatheca = <current metatheca head>,analyzer = <as supplied>,bm25 = <as supplied or default>,dictionary = dict,docstats = ds,created_at_ns = now(). Deposit it; letgbe the returned hash.Set the search-head pinax (§6) on the logopsis cella to
gwithexpected = None.
The genesis search-state contains no terms; it exists so that
reindex (§5.4) has a chain to advance from and
so that the analyzer identity and BM25 parameters are fixed at
init time.
On any failure prior to step 7, the implementation MUST leave no partially-initialized logopsis cella. If the cella was created in step 3, it MUST be removed before returning the failure.
5.2 Tokenize the corpus
Given a metatheca state M and an analyzer identity aid,
produce the analyzer output for every content blob reachable from
M:
Walk
M’s cumulative fact set (metatheca SPEC §3.3) and collect the set ofcore/blob-refblobs current in the projection (metatheca SPEC §5.5).For each such (entry, blob) pair
(e, b): a. Fetchbytes = syntheca.get(b). b. Compute the canonical token sequence with positionstokens(e, b) = analyze(bytes, aid)per §1.2. c. Recordlen(e, b) = |tokens(e, b)|.
The analyzer function is opaque to this specification. Its only
contract is determinism: identical input bytes under identical
aid MUST produce identical output token sequences and
position sequences.
Token sequences are not themselves deposited; they are consumed in-process by §5.3 to build the inverted index. A future phase may introduce tokenization caches (§11).
5.3 Build the inverted index
Given the tokenized corpus from §5.2, materialize the inverted index:
Build an in-memory map from each unique term
tto the ordered sequence of(entry, blob, tf, positions)records for every blob containingt. Records within a term’s sequence MUST be ordered byentry(UUID v7 ascending). Positions within a record MUST be ordered ascending.For each term
t, encode its posting list per §4.3 and deposit it; letp_tbe the returned hash anddf_tbe the record count.Construct the term-dictionary blob with one entry per term
t, containingt,df_t, andp_t, sorted by term ascending in UTF-8 byte order. Deposit it; letdictbe the returned hash.Construct the document-statistics blob with
doc_count,total_tokens,avgdl, andlengthsderived from §5.2. Deposit it; letdsbe the returned hash.Return
(dict, ds).
5.4 Reindex
Atomically advances search-head from the current search-state to a new search-state built against a chosen metatheca state.
Inputs: a metatheca stateref (default: metatheca’s current head).
Resolve the metatheca stateref per metatheca SPEC §5.3 to a metatheca state hash
M.Read the current search-state blob via the search-head pinax (§6); extract its analyzer identity
aidand BM25 parameters(k1, b). Compute the storage digestdof the search-head pinax bytes:sha256over the bytes returned bysyntheca.get_pinax, or remembered from the prior successfulset_pinax(apotheca SPEC §2.5/§3.5expectedsemantics).Tokenize the corpus per §5.2 against
Munderaid.Build the inverted index per §5.3; obtain
(dict, ds).Construct the new search-state blob:
previous = <current search-head>,metatheca = M,analyzer = aid,bm25 = {k1, b},dictionary = dict,docstats = ds,created_at_ns = now(). Deposit it; letsbe the returned hash.Call
syntheca.set_pinax(name = "head", bytes = s_bytes, expected = Some(d))against the logopsis cella.
The commit point is step 6. Steps 1–5 are pre-commit and produce only content-addressed deposita that are safe under interruption.
On Conflict from step 6, another writer advanced search-head
concurrently. The caller MAY retry from step 2.
The analyzer identity and BM25 parameters are fixed by the
genesis search-state and inherited unchanged by every subsequent
search-state. reindex MUST refuse to write a search-state
with an analyzer or bm25 field that differs from the
current search-state’s. Changing either is out of Phase 1 scope
(§10).
5.5 Query
Given a parsed query expression and a search-stateref, return the
top k documents by BM25 score.
5.5.1 Query expressions
A Phase 1 query expression is one of:
a single term, denoted
t;a phrase, denoted
"t1 t2 … tn", matching documents whose analyzer output contains the exact ordered sequence at adjacent positions;a conjunction, denoted
e1 AND e2, matching documents matching both subexpressions;a disjunction, denoted
e1 OR e2, matching documents matching either subexpression;a negation, denoted
NOT e, matching documents not matchinge.
AND is the implicit operator between adjacent expressions:
the query foo bar is equivalent to foo AND bar.
Parentheses MAY be used for grouping. Phrase delimiters are ASCII
double quotes; phrase contents are passed through the chain’s
analyzer like any other input, and matching is on the analyzer’s
token stream.
The operator keywords AND, OR, and NOT are
recognized case-sensitively, uppercase only; lowercase forms are
ordinary terms. Phrase delimiters are the escape hatch for searching
the uppercase words themselves: "AND" parses as a one-term
phrase, not an operator (what term it matches is then determined by
the chain’s analyzer, like any other phrase content).
Operator precedence, highest to lowest: parenthesized group, phrase, NOT, AND (including implicit), OR.
5.5.2 Evaluation
Resolve the search-stateref per §1.9 to a search-state hash
s. Fetch and parse the search-state blob; extract its analyzer identityaid, BM25 parameters(k1, b), dictionary hashdict, and docstats hashds.Run the query string through
aid’s analyzer to obtain query terms with their query-side positions (used only for phrase matching). Implementations MUST verify that the analyzer used for the query matchesaid; mismatch isAnalyzerMismatch.Fetch and parse the term dictionary. For each query term
t, resolve the dictionary entry. Terms absent from the dictionary contribute zero to scoring and zero to phrase matches.For each present term, fetch its posting list (§4.3). For phrase queries, intersect posting lists by
(entry, blob)and verify that the required positional offsets occur consecutively in the per-document position arrays.Combine the per-term candidate sets per the query’s boolean structure. The complement universe for negation is the set of documents recorded in the docstats
lengthsarray: within a larger expression,NOT ematches exactly the documents of that universe not matched bye(but see step 6 for queries with no positively-matched terms). For BM25 scoring, fetch the docstats blob (§4.6) and for each candidate document compute the BM25 score (§1.10) using the search-state’s(k1, b), the docstatsavgdland the document’slen, and the per-termdf(from the dictionary) andtf(from the posting-list record). A phrase contributes to scoring as its constituent terms, each scored independently per §1.10; the phrase acts as a match constraint only. Negated subexpressions contribute nothing to scoring.Return the top
kdocuments by descending score, each as(entry, blob, score). Ties in score MUST be broken by ascendingentry(UUID v7 order), so that result order is deterministic across implementations. NOT-only queries (queries with no positively-matched terms) MUST return an empty result set; they are not interpreted as "all documents minus the negated set."
Query is read-only: it MUST NOT modify any depositum or
pinax.
5.6 Resolve a search-state reference
Given a search-stateref (§1.9) and the current search-head, return the unique search-state hash it denotes. Algorithmically identical to metatheca SPEC §5.3 with the search-state chain substituted for the metatheca state chain.
5.7 Walk the search-state chain
Algorithmically identical to metatheca SPEC §5.4 with the search-state chain substituted.
5.8 Destroy
Removes the logopsis cella and all logopsis state from the vault.
Inputs: none.
If
<vault>/ext/logopsis/cella/is absent, fail withNotFound.Recursively remove the directory
<vault>/ext/logopsis/(including the cella and any extension-private auxiliary state under it).
Destroy is total: on completion, no logopsis state remains on the
vault. metatheca state and any sibling extension state are
unaffected. A subsequent lgops init (§5.1)
succeeds if and only if destroy completed.
Destroy is the supported mechanism for switching analyzer identity
or BM25 parameters (§10): destroy the chain,
then re-run lgops init with the desired identity. It is also
the recovery mechanism for a corrupted logopsis cella, since the
chain is wholly derivable from metatheca’s cella by reindexing.
Destroy MUST NOT touch <vault>/cella/,
<vault>/index.db, or any other entry in <vault>/ext/.
5.9 Auxiliary state
An implementation MAY maintain extension-private
auxiliary state — caches, projections, or other derived
structures that accelerate its own operation — under
<vault>/ext/logopsis/, outside the cella.
§5.8 already provides for the removal of any such
state; this section states the contract that makes it legitimate.
Observational neutrality. Auxiliary state MUST NOT change the observable behaviour of any protocol operation: a query answered from auxiliary state MUST return results identical to the same query evaluated from the wire-format blobs (§4) — the same hits, the same order, the same scores. In particular, substituting a different tokenizer or ranking function inside an acceleration structure violates the analyzer-identity contract (§1.2) and the scoring definition (§1.10), however convenient the substitute.
Disposability. Auxiliary state MUST be wholly derived: deleting it at any moment MUST NOT affect correctness, only performance. It is rebuilt or repaired only by the implementation that owns it.
Containment. Auxiliary state MUST NOT be written into any cella, referenced from any wire-format blob, or otherwise become interchange surface. Another implementation MUST be able to operate on the same vault while ignoring it entirely.
Freshness. An implementation SHOULD key auxiliary state to the hash of the search-state it reflects: content addressing makes staleness detection exact, and anything the auxiliary state cannot serve — historical search-states, a stale or damaged structure — falls back to the wire-format blobs.
Maintaining no auxiliary state is always conformant (§9); these constraints bind only implementations that choose acceleration.
The reference implementation maintains a SQLite projection at
<vault>/ext/logopsis/index.db, written by reindex from its in-memory index and keyed by the new search-state’s hash. Term and document reads become point lookups instead of parsing the monolithic dictionary and docstats blobs; queries against any other search-state, or with a stale or corrupt projection, silently take the wire-format path. Equivalence between the two paths is enforced by a test sweep over query shapes, result sizes, evaluation strategies, and historical states — the recommended conformance technique for any implementation that adds acceleration. Deployment analysis and measurements:SCALE.mdin this directory.
6 Search-head pinax
6.1 Name
The search-head pinax MUST be stored on the logopsis cella under
the apotheca pinax name head (four ASCII octets: h,
e, a, d). This name is fixed by this
specification.
The name is a valid apotheca name (apotheca SPEC §4.1):
non-empty, length 4, no /, no NUL. logopsis therefore imposes
no multi-segment-name requirement on the underlying apotheca
implementation.
The pinax shares only its name with metatheca’s head
(metatheca SPEC §6.1) and with any sibling extension’s search-head;
the pinakes live in disjoint cellae and never collide. The same
naming convention is used by semopsis (head on the semopsis
cella) and zetetes (head on the zetetes cella).
6.2 Bytes
The search-head pinax bytes follow metatheca SPEC §6.2 verbatim: the 64-octet ASCII encoding of the current search-state’s BLAKE3 hash as 64 lowercase hex digits, with no trailing newline or whitespace.
6.3 Compare-and-swap protocol
All updates to search-head MUST go through
syntheca.set_pinax with expected set to the SHA-256
digest of the current search-head bytes, identical in mechanism to
metatheca SPEC §6.3. The genesis case (init,
§5.1) sets search-head with expected = None.
7 CLI surface
The reference CLI binary is logopsis with the recommended
alias lgops. It exposes the protocol against the vault rooted
at the current working directory unless overridden with
--vault <PATH>.
This section specifies the protocol-bearing subcommands. Diagnostic output formatting is implementation-defined unless otherwise noted.
7.1 lgops init
lgops init --analyzer <family>:<version>:<language>
[--k1 <number>] [--b <number>]
[<path>]
Initialize a logopsis chain on a vault per §5.1. The
vault root is <path> if supplied, otherwise the current
directory. The vault MUST already contain a metatheca chain.
Exits non-zero if metatheca is absent or a logopsis chain already
exists.
--k1 defaults to 1.2 and --b defaults to
0.75 per §1.10. Implementations SHOULD warn
when defaults are used so the caller is aware these are now
permanent for the chain.
7.2 lgops reindex
lgops reindex [--as-of <metatheca-stateref>]
Build a new search-state per §5.4 against the named metatheca state (default: metatheca’s current head). On success, prints the new search-state hash to standard output.
7.3 lgops query
lgops query [--k <int>] [--as-of <search-stateref>] <query>
lgops query [--k <int>] [--as-of <search-stateref>] -
Run a query per §5.5. <query> is the query
string in the syntax of §5.5.1; - reads the
query from standard input. Default --k is 10. Default
--as-of is current.
Output format per result, one line:
<entry-uuid> <blob-hash> <score>
Scores are formatted as IEEE 754 binary64 in %.6g form.
7.4 lgops state
lgops state [<search-stateref>]
Show metadata for a search-state: hash, created_at_ns,
previous, metatheca, analyzer identity, BM25
parameters, dictionary term count, total document count. Default
argument: current.
7.5 lgops log
lgops log [--since <iso8601>] [--until <iso8601>] [-n <count>]
Walk the search-state chain from current search-head backward, emitting one line per search-state with its short hash, ISO-8601 timestamp, analyzer identity, and metatheca state short hash.
7.6 lgops fsck
lgops fsck
Verify: the search-head pinax bytes encode a search-state present
in the cella; every search-state in the chain parses; the
analyzer identity and BM25 parameters are constant across the
chain; every term-dictionary parses, its terms array is
sorted ascending in UTF-8 byte order with no duplicates, and every
referenced posting list is present and parses; every posting
list’s record count equals the dictionary’s df for the
corresponding term; every posting list’s records are sorted by
entry ascending with positions ascending within each record;
every docstats blob parses and its avgdl satisfies the
§4.6 rule (total_tokens / doc_count to
binary64 equality, 0 when doc_count = 0); every
search-state’s metatheca field names a metatheca state
present in the cella. Implementations MAY rely on syntheca and
apotheca verification for byte-level integrity.
7.7 lgops destroy
lgops destroy [--force] [<path>]
Destroy the logopsis cella per §5.8. The vault root
is <path> if supplied, otherwise the current directory.
Unless --force is supplied, implementations SHOULD prompt
for confirmation before proceeding, since destroy is irreversible
without a metatheca-side reindex (§5.4) to rebuild
the chain. Exits non-zero if no logopsis cella exists at the
resolved path.
7.8 Reserved
The verb gc is reserved for future phases.
8 Errors
The error conditions defined by this specification are:
NotFound— a search-stateref or referenced blob did not resolve.Ambiguous— a hash-prefix search-stateref matched more than one search-state.Conflict— search-head CAS lost a race; the caller MAY retry (§5.4).Malformed— a wire-format blob did not parse per §4 (including invalid UTF-8 in term strings, posting-list ordering violations, dictionary sort violations, or docstats arithmetic mismatches), or a query string did not parse per §5.5.1, or an argument did not parse per §1.9.AnalyzerMismatch— a query was issued against a search-state whose analyzer identity differs from the analyzer used to tokenize the query (§5.5), or a reindex attempted to change the analyzer or BM25 parameters fixed at genesis (§5.4).IntegrityError— inherited from syntheca / apotheca; surfaces unchanged.
Implementations MAY surface additional implementation-defined errors (I/O failure, analyzer unavailable, …); such errors MUST be distinguishable from the protocol-defined errors above.
9 Conformance
A Phase 1 implementation MUST implement:
§2 (host/extension relationship)
§3 (identifiers, time, strings)
§4 (wire formats)
§5.1–§5.8 (operations), including analyzer and BM25-parameter pinning and inheritance, and the destroy operation
§5.9 (auxiliary state), applicable only when an implementation maintains such state
§6 (search-head pinax)
§7 (CLI), with the exception that
--as-ofMAY be omitted fromlgops queryif the implementation always queriescurrent§8 (errors)
A Phase 1 implementation MAY omit:
§10 items.
10 Out of scope (Phase 1)
The following are deliberately out of Phase 1 scope. Their absence is load-bearing for the v0.1 surface and consumers MUST NOT rely on them.
Analyzer migration. A search-state’s analyzer identity and BM25 parameters are immutable across the chain. Switching either requires destroying the logopsis chain (§5.8) and re-running
lgops init. A future phase may define rotation operations that preserve the chain.Incremental reindex. Phase 1 reindex rebuilds all posting lists from scratch. Content-addressing means unchanged posting lists deduplicate at the cella, but the work of recomputing postings is full each time.
Posting-list compression. Phase 1 stores positions as raw ULEB128 absolute offsets. Standard inverted-index compression techniques (delta encoding, PFOR, FastPFOR, roaring bitmaps for doc IDs, bit-packed positions) are deferred to a future phase. Compression changes the wire format and is therefore a versioned change, not an additive one.
Wildcard, prefix, and regex queries. Phase 1 matches whole terms only. Prefix and wildcard matching requires either a separate term-prefix index or n-gram indexing, neither of which is specified.
Fuzzy matching. Phase 1 has no edit-distance or phonetic-match support. Typo tolerance is a caller concern or a future-phase feature.
Field-scoped search. Phase 1 indexes blobs as opaque token sequences. There is no concept of a "title field" or "body field." If callers need field-scoped search, they should index field projections as separate logical entries at the metatheca layer.
Highlighting and snippets. Phase 1 returns matching documents and scores. Generating highlighted snippets from the original content is a caller concern.
Garbage collection. Old search-states and their referenced posting lists, dictionaries, and docstats remain on the cella indefinitely. Blocked by absence of delete in apotheca and syntheca.
Cross-vault index sharing. As semopsis §10.
Hybrid keyword+vector ranking. Hybrid retrieval combining logopsis’s lexical results with semopsis’s semantic results is the responsibility of the zetetes coordinator layer; it is not part of the logopsis protocol.
Multi-vector / multi-query batched evaluation. Single query expression per call.
11 Future work (non-normative)
Posting-list compression. A Phase 2 posting-list format with delta-encoded doc IDs (UUID v7’s monotonicity makes deltas small), bit-packed positions, and skip pointers for sub-linear conjunctive evaluation. The dictionary entry would gain a
formatdiscriminator so old and new posting lists can coexist on the same cella during migration.Tokenization caches. A
core/token-stream-reffact in metatheca, recording the BLAKE3 hash of an analyzed-token-stream depositum alongsidecore/blob-ref. This would let logopsis reuse tokenizations across reindexes without re-running the analyzer when the underlying blob is unchanged. Additive change to metatheca, parallel to semopsis’s proposedcore/embedding-ref.Dictionary and docstats sharding. Phase 1 materializes the term dictionary (§4.5) and the document statistics (§4.6) as one blob each per search-state, so query evaluation parses O(vocabulary) and O(corpus) bytes regardless of query size. Implementations can cache the parsed structures keyed by blob hash — content addressing makes such a cache exact — which amortizes the cost across queries in a long-lived process; the blobs themselves become the binding constraint only at corpora well beyond the personal-vault envelope (order 10^5 documents and up). A Phase 2 format could shard the dictionary by term prefix and the docstats by entry range behind a manifest blob, with a format discriminator so old and new layouts coexist on a cella during migration, mirroring the posting-list compression path above.
Word-shingle (n-gram) indexing — considered, not planned. Indexing adjacent word pairs and triples as dictionary terms would turn quoted-phrase evaluation into a single posting-list lookup. Measured against the Rust reference implementation (2026-07-03,
examples/phrase_bench.rs, a uniform 24-word vocabulary making every constituent stopword-frequent — the adversarial ceiling): a quoted phrase costs about the same as the equivalent conjunctive query in every corpus shape (many small documents, few huge documents, both), because the cost is decoding the constituent posting lists — which any multi-term query pays — while the positional verification itself is nearly free. Worst-case phrase-attributable cost was ~30 ms at 10^7 tokens; where query latency was dominated at all, the dominator was the monolithic dictionary/docstats parse (see the sharding entry above), which shingle indexing would inflate — distinct shingles in natural-language corpora grow near-linearly with corpus size, and each indexed n adds roughly one further copy of the positions data. Revisit only for corpora around 10^8 tokens and up, after dictionary sharding; the natural shape then is an optional shingle side-index at declared lengths, format-discriminated like the compression path.Skip-list pointers in the search-state blob, mirroring metatheca SPEC §11 and semopsis SPEC §11, to make as-of resolution sublinear.
Phase 2 analyzer rotation: a new search-state kind that records both an old and a new analyzer identity, allowing graceful migration without destroying the chain. Useful for Unicode-version updates, stopword-list revisions, and stemmer upgrades.
Phase 2 BM25-parameter rotation: analogous, for
(k1, b)retuning. Less commonly needed than analyzer rotation but specifiable with the same machinery.Phase 2 incremental reindex: posting-list-level deltas keyed by term identity, with merge passes to reconsolidate. Likely pairs with the compression work above to be useful.
Field-scoped indexing as a Phase 2 feature, with each entry carrying a sparse
(field, blob, tokens)structure rather than a single token stream. Touches the dictionary, the posting-list format, and the docstats simultaneously; non-trivial.A remote-cella backend (apotheca §10) makes the per-term-posting-list layout natural for object-storage deployment. The query-time fan-out is bounded by the number of distinct query terms (typically 1–10), not by corpus size, so per-member embedding-pack-style optimizations are unnecessary; logopsis on R2 is naturally well-suited to object-storage deployment.
A remote query service. The remote-cella deployment above still makes the client pay the query fan-out over the internet; a companion serving tier (hyperetes, the servant) that evaluates §5.5.2 beside the cella and returns only the top-
kmakes the client cost independent of corpus size. It owns no chain state and is not an extension — pure serving infrastructure over the query surface. The constraint it feeds back into this spec: Phase 2 formats and query cores should stay I/O-agnostic (byte-range reads behind an async seam) and compilable to WASM, so a conformant evaluator can run in an edge runtime at zero distance from the cella. Deployment analysis and measurements:SCALE.mdin this directory, scenario 4.