apotheca Specification

1. Scope and terminology

1.1 Cella

A cella is the unit of storage. It holds an unbounded set of named deposita, each depositum consisting of a name and the bytes stored under that name. A cella is composed of one or more backends; this document specifies the single-backend case in full and defers multi-backend composition to a future revision (see §10).

1.2 Depositum

A depositum is a (name, bytes) pair durably held by a cella. The term is geological: like sedimentary deposita, each depositum is laid down once and is thereafter neither modified nor removed. Once a deposit operation returns Ok, the bytes under that name are immutable for the lifetime of the cella. A depositum is present in a cella once its deposit operation has returned Ok and the cella has not since been corrupted or destroyed.

“Entry” is permissible as an informal English synonym in prose; normative text in this specification uses depositum exclusively.

1.3 Name

A name is a caller-chosen identifier under which bytes are stored. Name constraints are defined in §4.

1.4 Bytes

Bytes denote an arbitrary octet sequence of length in [0, 2^63). The zero-length sequence is a valid value.

1.5 Digest

A digest in this specification is a SHA-256 hash, represented as 32 octets. SHA-256 is defined by FIPS 180-4.

1.6 Pinax

A pinax is a (name, bytes) pair held by a cella whose bytes MAY be replaced by a subsequent successful set_pinax. A cella's pinakes are kept in a namespace disjoint from its deposita: the same name MAY be used for both a pinax and a depositum without collision (see §4.3). The term is borrowed from Greek πίναξ (pinax), tablet or list-board; the plural is pinakes. Where a depositum is sediment laid down once and never disturbed, a pinax is a surface designed to be overwritten.

A pinax is present in a cella once any set_pinax for that name has returned Ok and the cella has not since been corrupted or destroyed.

Pinakes are intended for small, frequently-updated values such as state pointers and history heads. The spec imposes no maximum size; implementations SHOULD support pinakes of at least 1 KiB and MAY reject larger pinax bytes with an implementation-defined error.

2. Operations

A cella exposes five operations: deposit, get, and stat on the depositum namespace; get_pinax and set_pinax on the pinax namespace. Each operation is total: it terminates with one of the outcomes listed for it.

2.1 deposit(name, bytes)

Stores bytes under name.

• Ok — name was absent, or was present with bytes whose digest equals sha256(bytes). On the absent case, bytes are now stored under name (see §5 for atomicity). On the equal case, the stored bytes are unchanged (idempotent re-deposit).
• Collision — name is present with bytes whose digest differs from sha256(bytes). The stored bytes MUST NOT be modified.

The implementation MAY compute sha256(bytes) directly, or MAY compare the candidate bytes with the stored bytes by any means that yields the same outcome.

2.2 get(name)

Returns the bytes stored under name. Implementations MUST verify the returned bytes against the stored digest before returning them to the caller.

• Bytes — name is present and verification succeeds. The returned bytes are equal, octet-for-octet, to the bytes most recently passed to a deposit call that returned Ok for name.
• NotFound — name is not present.
• IntegrityError — name is present, but the bytes read from the backend have a digest that does not equal the stored digest. The bytes MUST NOT be returned to the caller.

IntegrityError indicates backend corruption (silent disk error, partial write surviving recovery, tampering). It is a real condition, not a theoretical one; callers SHOULD propagate it as an error rather than retry.

2.3 stat(name)

Returns metadata for name without reading the bytes.

• Metadata = { size: octet count in [0, 2^63), sha256: 32-octet digest } — name is present.
• NotFound — name is not present.

stat MUST be implementable without transferring the bytes. On backends where bytes and metadata are co-located (e.g. local filesystem), implementations MAY return metadata derived from on-disk state without re-hashing on every call.

2.4 get_pinax(name)

Returns the bytes stored under name in the pinax namespace. Implementations MUST verify the returned bytes against the stored digest before returning them to the caller (see §3).

• Bytes — name is present and verification succeeds. The returned bytes are equal, octet-for-octet, to the bytes most recently passed to a set_pinax call that returned Ok for name.
• NotFound — name is absent in the pinax namespace.
• IntegrityError — name is present, but the bytes read from the backend have a digest that does not equal the stored digest. The bytes MUST NOT be returned to the caller.

2.5 set_pinax(name, bytes, expected)

Stores bytes under name in the pinax namespace, conditionally on the current value matching expected. The argument expected is Option<Digest>:

• expected = None requires name to be absent in the pinax namespace.
• expected = Some(d) requires name to be present with stored digest equal to d.

Outcomes:

• Ok — the precondition held; bytes are now stored under name, replacing any prior value (see §5.4 for atomicity). If the new bytes' digest equals the stored digest of an already-present pinax, the call is idempotent and the stored bytes are unchanged.
• Conflict { actual: Option<Digest> } — the precondition did not hold. actual = None reports that name was absent; actual = Some(d') reports that name was present with stored digest d'. The stored bytes MUST NOT be modified.

The actual digest reported with Conflict MUST be observed by the same atomic check that determined the precondition failed. Implementations SHOULD return Conflict to the caller rather than retry internally; the standard compare-and-swap pattern is for the caller to retry with actual as the new expected.

The implementation MAY compute sha256(bytes) directly, or MAY compare with the stored bytes by any means that yields the same outcome.

3. Integrity

3.1 Mandatory hashing

Every present depositum MUST have a stored sha256 digest. There is no mode in which deposita are stored without an associated digest.

3.2 Verification

get MUST verify; deposit MUST detect collisions by digest equivalence; stat MUST report the stored digest. The digest is the only quantity that defines “same bytes” or “different bytes” for the purposes of this specification.

3.3 ETag semantics

The stored sha256 IS the depositum's ETag. Backends that have a native ETag-like field (e.g. S3-compatible object stores) MUST place the apotheca digest in a field whose value is exactly sha256(bytes) (e.g. the x-amz-checksum-sha256 header on S3-compatible backends, not the legacy multipart-derived ETag). Native ETags whose value is not sha256(bytes) MUST NOT be used to satisfy this specification.

3.4 Hash function

The hash function is SHA-256 and is fixed by this specification. Changing the hash function is a protocol revision.

3.5 Pinakes

The integrity guarantees of §3.1–§3.4 apply equally to pinakes: every present pinax MUST have a stored sha256 digest; get_pinax MUST verify; set_pinax MUST detect equivalence by digest; the stored sha256 IS the pinax's ETag for backends with native ETag-like fields (§3.3). The local backend MAY derive the stored digest from the stored bytes on demand rather than persisting it in a separate file, provided every observable read returns a digest identical to what would be returned by a backend that stores it separately.

4. Name policy

4.1 Names

A name is a single non-empty filesystem-safe component:

• non-empty
• contains no path separator (/)
• contains no NUL octet
• is not . or ..
• length in octets is in [1, 255]

Names are octet sequences, not Unicode strings. Implementations MUST NOT apply Unicode normalisation. Two names are equal iff their octet sequences are equal.

4.2 Future names

Future revisions MAY admit names containing /, with each /-separated segment subject to the §4.1 component constraints. Implementations SHOULD treat the namespace as opaque: foo/bar and foo have no hierarchical relationship as far as the protocol is concerned.

4.3 Pinakes and deposita

Pinakes and deposita are kept in disjoint namespaces within a cella. The same name MAY refer to a pinax and a depositum simultaneously; operations on the depositum namespace (§2.1–§2.3) are independent of operations on the pinax namespace (§2.4–§2.5). Name constraints (§4.1) apply identically to both namespaces.

5. Atomicity

5.1 All-or-nothing visibility

A successful deposit operation MUST be all-or-nothing with respect to readers: a concurrent get or stat MUST observe either the full prior state of the name (including absence) or the full new depositum (bytes plus digest). No intermediate state in which bytes are present without their digest, or vice versa, is observable.

5.2 Crash safety

After a process crash, host crash, or power loss, the cella MUST be in a state consistent with §5.1: every name that is observable as present MUST have its bytes and its stored digest both intact and matching. Implementations MAY discard partially-written deposita during recovery.

5.3 Concurrent writers

Concurrent deposit calls for the same name MUST resolve such that exactly one of the following holds:

• both writers passed bytes with the same digest, and both observe Ok;
• writers passed bytes with differing digests, and at least one observes Collision; the depositum under name after both calls return is one of the digests submitted, and is the digest reported by every subsequent stat and the digest of the bytes returned by every subsequent get.

5.4 Pinax atomicity

A successful set_pinax MUST be all-or-nothing with respect to readers: a concurrent get_pinax MUST observe either the full prior value (including absence) or the full new value. No intermediate state in which bytes and digest disagree, or in which the pinax transiently appears absent during a replacement, is observable.

The precondition check against expected and the write are a single atomic step. Concurrent set_pinax calls for the same name MUST be linearised: each call observes the result of all calls preceding it. If two calls would store differing bytes under the same expected, at most one MAY observe Ok; the others MUST observe Conflict whose actual reflects either the prior state or the state set by the call that succeeded.

After a process crash, host crash, or power loss, every pinax observable as present MUST have its bytes and stored digest both intact and matching. Implementations MAY discard partially-written pinax staging files during recovery (§6.5).

6. Local backend on-disk layout

This section specifies the on-disk layout of the local-filesystem backend.

6.1 Cella root

The cella root is a directory. The default cella root is ~/.apotheca/. The root MAY be overridden by configuration; the layout below is relative to it.

6.2 Layout

<root>/
  deposita/
    <name>/
      bytes
      meta
  tmp/
    <staging-id>/
      bytes
      meta

• deposita/<name>/bytes contains the depositum's bytes, octet-for-octet.
• deposita/<name>/meta contains the depositum's metadata in the format defined in §6.3.
• tmp/<staging-id>/ is a staging area used to satisfy §5.

6.3 Meta file format

The meta file is UTF-8 text, exactly:

size <decimal>
sha256 <hex>

with one trailing newline after the sha256 line. <decimal> is the unsigned decimal representation of the depositum's size in octets. <hex> is the depositum's sha256 as 64 lowercase hexadecimal digits.

Implementations MUST reject meta files that do not match this grammar exactly.

6.4 Atomic deposit procedure

A conforming local-backend deposit(name, bytes) proceeds as follows:

1. Compute d = sha256(bytes).
2. If deposita/<name>/ exists, parse deposita/<name>/meta. If its sha256 field equals d, return Ok. Otherwise return Collision.
3. Choose a fresh <staging-id> (e.g. a random component) and create tmp/<staging-id>/.
4. Write bytes to tmp/<staging-id>/bytes; fsync the file.
5. Write the meta file to tmp/<staging-id>/meta; fsync the file.
6. Fsync the staging directory.
7. Atomically rename tmp/<staging-id>/ to deposita/<name>/ (rename(2)).
8. Fsync the parent of deposita/<name>/.

Step 7 is the linearisation point: before it, no reader observes the new depositum; after it, every reader observes both bytes and meta together.

If any step fails, the staging directory MAY be removed; if it is left behind, recovery (§6.5) handles it.

6.5 Recovery

On startup, or on demand, an implementation MAY scan tmp/ and remove staging directories older than an implementation-defined threshold. tmp/ content is never authoritative; deposita become authoritative only when renamed into deposita/.

6.6 Get and stat procedures

get(name):

1. If deposita/<name>/ does not exist, return NotFound.
2. Parse deposita/<name>/meta to obtain (size, sha256_stored).
3. Read deposita/<name>/bytes.
4. If the read length does not equal size, return IntegrityError.
5. Compute sha256(bytes_read). If it does not equal sha256_stored, return IntegrityError.
6. Return the bytes.

stat(name):

1. If deposita/<name>/ does not exist, return NotFound.
2. Parse deposita/<name>/meta to obtain (size, sha256_stored).
3. Return { size, sha256_stored }.

stat does not re-hash and does not read bytes.

6.7 Pinax layout

Pinakes are stored under <root>/pinakes/, parallel to and disjoint from <root>/deposita/:

<root>/
  deposita/                       # depositum namespace, write-once (§6.2)
  pinakes/                        # pinax namespace, compare-and-swap
    <name>                        # one regular file per pinax; content = bytes
    <name>.lock                   # per-name lockfile (created on demand)
  tmp/                            # staging area, shared (§6.2)
    <staging-id>

Each pinax is stored as a single regular file at <root>/pinakes/<name> whose content is the pinax's bytes. The local backend recomputes the stored digest from the file content on each read (§3.5). No meta file is written for pinakes.

This layout permits atomic file-over-file replacement via rename(2), which is portable and crash-safe; the per-name-directory layout used for deposita (§6.2) does not admit a portable atomic-replace primitive on POSIX.

6.8 Atomic set_pinax procedure

A conforming local-backend set_pinax(name, bytes, expected) proceeds as follows. The procedure MUST hold an exclusive advisory lock on <root>/pinakes/<name>.lock (created on demand) for the duration of steps 2–7.

1. Compute d = sha256(bytes).
2. Acquire the exclusive lock.
3. Determine the current state of <root>/pinakes/<name>:
   • if it does not exist, set actual = None;
   • else read the file content and set actual = Some(sha256(content)).
4. If actual != expected, release the lock and return Conflict { actual }.
5. If actual == Some(d) (idempotent re-set with identical bytes), release the lock and return Ok without writing.
6. Otherwise, choose a fresh <staging-id>, write bytes to tmp/<staging-id>, fsync the file.
7. Atomically rename tmp/<staging-id> to <root>/pinakes/<name> (rename(2) over the existing file is atomic on POSIX). Fsync <root>/pinakes/.
8. Release the lock and return Ok.

Step 7 is the linearisation point. Readers (get_pinax) do not take the lock and observe whichever rename is current.

If any step after step 6 fails, the staging file is orphaned and recovery (§6.5) handles it.

6.9 get_pinax procedure

get_pinax(name):

1. If <root>/pinakes/<name> does not exist, return NotFound.
2. Read the file content as bytes_read.
3. Compute sha256(bytes_read) and return bytes_read together with this digest as the verification.

The local backend, having derived the digest from the bytes just read, cannot observe an integrity mismatch within this procedure. The IntegrityError outcome of §2.4 is reachable only on backends that store the digest separately from the bytes (e.g. S3-compatible, where the digest is a header) and observe a mismatch between the stored and recomputed digests.

7. CLI surface

The reference CLI binary is apo. It exposes the protocol operations one-for-one, against the default local cella (§6.1) unless configured otherwise.

7.1 apo deposit

apo deposit [--name <name>] <path>
apo deposit  --name <name> -

• <path> reads bytes from the named file. With <path> and no --name, the name defaults to the basename of <path>. With --name, the given name is used.
• - reads bytes from standard input. --name is REQUIRED in this form.
• Exit status 0 on Ok (including idempotent re-deposit). Exit status non-zero on Collision or any I/O error, with a diagnostic to standard error.

7.2 apo get

apo get <name>

• Writes the bytes for <name> to standard output. Performs verification per §2.2.
• Exit status 0 on success. Exit status non-zero on NotFound, IntegrityError, or any I/O error, with a diagnostic to standard error and no partial output if the failure can be detected before writing.

7.3 apo stat

apo stat <name>

• Writes a human-readable summary of the metadata to standard output, in the form:

  size   <decimal>
  sha256 <hex>

• Exit status 0 on success. Exit status non-zero on NotFound or any I/O error.

7.4 apo pinax get

apo pinax get <name>

• Writes the bytes for the pinax <name> to standard output. Performs verification per §2.4.
• Exit status 0 on success. Exit status non-zero on NotFound, IntegrityError, or any I/O error, with a diagnostic to standard error.

7.5 apo pinax set

apo pinax set --name <name> (--expect-absent | --expect <hex>) <path>
apo pinax set --name <name> (--expect-absent | --expect <hex>) -

• <path> reads bytes from the named file; - reads from standard input.
• Exactly one of --expect-absent and --expect <hex> MUST be given. --expect-absent corresponds to expected = None; --expect <hex> corresponds to expected = Some(<hex>) where <hex> is 64 lowercase hexadecimal digits.
• Exit status 0 on Ok (including idempotent re-set). On Conflict, exit status non-zero and a diagnostic of the form conflict: actual=<hex> or conflict: actual=absent written to standard error. Other I/O errors exit non-zero with a diagnostic.

8. Errors

The error conditions defined by this specification are:

• Collision (§2.1) — deposit operation with differing digest under existing name.
• Conflict (§2.5) — pinax set_pinax precondition failed.
• NotFound (§2.2, §2.3, §2.4) — name absent in the relevant namespace.
• IntegrityError (§2.2, §2.4) — stored bytes do not match the stored digest.

Implementations MAY surface additional implementation-defined errors (I/O failure, permission denied, malformed meta file, exhausted disk space, …). Such errors MUST be distinguishable from the protocol-defined errors above; in particular, an I/O failure MUST NOT be reported as NotFound, Collision, or Conflict.

9. Conformance

9.1 Depositum-surface conformance

An implementation conformant to the depositum surface MUST implement:

• §1.1–§1.5 (terminology for deposita)
• §2.1–§2.3 (deposit operations: deposit, get, stat)
• §3.1–§3.4 (depositum integrity)
• §4.1 (Names)
• §5.1–§5.3 (depositum atomicity)
• §6.1–§6.6 (local backend layout, atomic deposit procedure, recovery, get/stat for deposita)
• §7.1–§7.3 (CLI for deposita)
• §8 (errors), restricted to Collision, NotFound, and IntegrityError

Such an implementation MAY omit:

• §4.2 (multi-segment names) — names with / are rejected.
• §10 (multi-backend composition).
• All pinax-related subsections (the pinax surface — see §9.2).

9.2 Pinax-surface conformance

An implementation conformant to the pinax surface MUST implement every requirement of the depositum surface (§9.1), plus:

• §1.6 (pinax terminology)
• §2.4, §2.5 (get_pinax, set_pinax)
• §3.5 (pinax integrity)
• §4.3 (disjoint pinax/depositum namespaces)
• §5.4 (pinax atomicity)
• §6.7–§6.9 (local backend pinax layout and procedures)
• §7.4, §7.5 (CLI pinax subcommands)
• §8 (errors), additionally including Conflict

A depositum-surface implementation MAY ship pinax-surface features additively without claiming pinax-surface conformance, provided its depositum-surface conformance remains intact.

10. Future work (non-normative)

The following are explicitly out of scope for v1.0-rc1 and will be specified in future revisions:

• Additional backend kinds (S3-compatible, scp/sftp, read-only HTTP).
• Multi-backend cellae, write fan-out semantics, read priority.
• Encryption-as-wrapper backend.
• Multi-segment names (§4.2).
• Configuration mechanism (cella root override, per-cella backend lists).

These are listed here so that implementations and consumers know not to rely on their absence.