Documents API

Manage and query documents within the Aether Object Store using the Aether SDK.

Documents are the foundation of Aether. Before you can search for anything, you need to store your content — PDFs, text files, markdown, or raw strings. When you insert a document, Aether automatically splits it into chunks, generates vector embeddings for each chunk, and indexes everything for search. You don't need to worry about the chunking or embedding process — it happens behind the scenes.

SDK methods

Every document operation is a method on the AetherClient. Names follow each language's conventions; full usage for each is shown in the examples below this table.

¹ The .NET SDK has no text helper — decode the bytes from DownloadAsync, e.g. Encoding.UTF8.GetString(bytes).

Your API key is set once when you construct the client (see Authentication); the SDK attaches it to every request automatically. Failed calls raise an SDK error that maps to the status codes documented in Errors.

Document route contract

The SDK methods above wrap the hosted REST contract. Use the SDK for application code; use this route table when you are debugging, writing a proxy, or checking the wire shape.

Direct REST requests authenticate with Authorization: Bearer <api-key>. SDK users should not set this header manually — construct the client with an API key and let the SDK attach it.

API versioning

The canonical REST surface is versioned under the /v1 prefix — for example https://api.aetherdb.ai/v1/documents. Every data route (/v1/documents*, /v1/threads*, /v1/search*, /v1/memory/*, /v1/partitions*) follows this scheme.

Unversioned paths (e.g. /documents) keep working as deprecated aliases: they return the same responses, plus a Deprecation: true header and a Sunset: Fri, 01 Jan 2027 00:00:00 GMT header. They will be removed after January 1, 2027 — if you call the REST API directly, move to /v1 before then. SDK users don't need to do anything; the SDKs track the canonical paths.

One exception: GET /status, the public health probe, is intentionally unversioned.

Chunking and CIDs

Aether stores documents as content-addressed chunks:

• The default storage chunk size is 256KB.
• Insert and update calls can pass chunk_size; SDKs expose it as chunk_size, chunking.chunkSize, ChunkingConfig.ChunkSize, or aether.WithChunkSize(...) depending on language.
• overlap is optional and defaults to 0 when a custom chunk_size is used.
• Synchronous HTTP requests are capped at 50MB by the API layer. The node also enforces its configured max_document_bytes limit before work is accepted. Use async ingestion for parser-backed formats and long-running uploads.
• Each stored chunk gets a deterministic CID built from a BLAKE3 hash of the chunk bytes and encoded with the aether: prefix.

The top-level document cid in a DocumentRecord is stable for identical content. Chunk CIDs let Aether verify and route stored bytes across swappable storage backends without depending on filenames or database row ids.

Supported file types

Text and office formats are extracted and indexed automatically. Sending an unsupported format returns 422.

Interacting with Documents

The AetherClient makes accessing documents simple.

Inserting Documents

There are two ways to add content. insert_text is the quickest way to get started — just pass a string directly and Aether takes care of the rest. Use insert when you have a file path or binary data (like a PDF or an uploaded file buffer). Both methods return a DocumentRecord that tells you the assigned doc_id, how many chunks the content was split into, and how many vectors were generated. This is useful for verifying that your document was processed correctly — if a document has zero chunks, something likely went wrong with the input.

# Insert from a path
doc = client.insert(
    file_path="./data/manifesto.md", 
    content_type="text/markdown"
)

# Insert raw text string directly
doc_text = client.insert_text(
    text="Some text in the object store.",
    filename="text.txt"
)

print(doc_text.doc_id, doc_text.chunks)

Both methods return a DocumentRecord containing the doc_id, cid (Content Identifier), the document's tags and structured metadata, and counts for the chunks and vectors generated.

Structured metadata

Use metadata for typed fields you want to filter later. Metadata values must be strings, numbers, or booleans — or a flat array of strings, for a tag-like value with several entries. Use RFC 3339 strings for timestamps you want to compare chronologically. Nested objects, nested arrays, non-string array elements, and null are rejected.

A string array is what a string_list field lifts, so contains matches any single element. Scalar metadata filters never match an array-valued key — an array equals no scalar — so membership tests go through contains on a declared field, or through tags.

doc = client.insert_text(
    "Session notes: intake visit, clinical severity 0.8.",
    filename="session-1.txt",
    metadata={
        "severity": 0.8,
        "is_clinical": True,
        "session_type": "intake",
    },
)

print(doc.metadata["severity"])

update replaces a document's structured metadata. Pass the full metadata map you want to keep, just as you do with tags. Query examples live in Search & Retrieval.

Quotas & Plan Limits

As a fully managed service, Aether enforces storage and throughput bounds based on your current organization's subscription tier. If an insertion or embedding request exceeds your permitted quota (e.g., maximum storage bytes, or maximum active documents), the API will reject the request with a 402 Plan limit exceeded status. Your applications should be prepared to handle these 402 responses or monitor your usage periodically via the Dashboard to avoid unexpected ingestion interruptions.

Streaming Large Files

When inserting multi-gigabyte files, loading the entire payload into RAM before transmission can lead to memory exhaustion. To prevent this, the SDKs provide an insert_stream method that streams the binary data directly to Aether's load balancers.

# Pass a file object opened in binary mode
with open("./data/massive-knowledge-base.zip", "rb") as f:
    doc = client.insert_stream(
        stream=f,
        filename="massive-knowledge-base.zip",
        content_type="application/zip"
    )
print(doc.doc_id)

Async Ingestion

For large files or batch uploads, use async ingestion to avoid blocking. insert_async returns immediately with a job handle (job_id, status, poll_url); wait_for_job then polls until the job reaches a terminal state — completed or failed — and returns the final status (doc_id on success, error on failure). It raises a timeout error if the job doesn't finish within the deadline (60 s by default).

# Start an async insert — returns immediately with {job_id, status, poll_url}
job = client.insert_async(file_path="./data/large-report.pdf", content_type="application/pdf")
print(f"Job started: {job['job_id']}")

# Block until the job reaches a terminal state (polls in the background)
result = client.wait_for_job(job["job_id"])

if result["status"] == "completed":
    print(f"Document ready: {result['doc_id']}")
else:
    print(f"Job failed: {result.get('error')}")

The raw job endpoint returns a progress snapshot:

{
  "job_id": "8d3f1c26-9ea1-4ec9-b701-6e3f1c2a7a01",
  "status": "embedding",
  "doc_id": null,
  "message": "Embedding document",
  "progress": 0.6,
  "created_at": "2026-06-25T18:30:00Z",
  "finished_at": null,
  "error": null
}

Terminal states are completed and failed. Jobs are tenant-scoped; polling a job from another tenant returns 404 rather than leaking that it exists.

Custom Embeddings (BYOE)

If you use your own embedding model, you can insert documents with pre-computed vectors instead of relying on Aether's built-in embedder. Provide either a single embedding for the whole document, or per-passage embeddings for fine-grained retrieval.

# Insert with per-passage embeddings
doc = client.insert_with_embeddings(
    content="Full document text...",
    filename="report.md",
    passages=[
        {"text": "First paragraph...", "embedding": [0.1, 0.2, ...]},
        {"text": "Second paragraph...", "embedding": [0.3, 0.4, ...]},
    ]
)

Updating and Retrieving Metadata

Sometimes you need to replace a document's content — maybe you fixed a typo, or a newer version of the file is available. Calling update replaces the content entirely: Aether re-chunks the new content, generates fresh embeddings, and re-indexes everything. The old vectors are discarded. On the other hand, get is a lightweight call that just returns metadata (like chunk count, file size, and content type) without downloading the actual content. It's a quick way to check on a document without pulling the full payload. Document records also carry timestamps: get returns both created_at and updated_at, and each item from list includes created_at.

You can update the actual contents of an existing document or fetch its metadata details.

# Overwrite an existing document by doc_id
updated = client.update("doc_12345", "./data/revised_manifesto.md")

# Simply fetch the latest metadata about the document
meta = client.get("doc_12345")
print(f"Chunks: {meta.chunks}, Size: {meta.size_bytes}")

Re-ingesting the Same Source

Send an Idempotency-Key header with POST /v1/documents and a retry stops being a duplicate. Use any stable string your own system already has — the upstream record id, a job id, a hash of the content:

The key commits the whole request: filename, content type, tags, entity_id, source, metadata, chunking, and the content itself. A retry that just reorders JSON keys is still the same request. Keys are scoped per workspace and per partition, so the same key in another partition is a separate document.

curl -X POST "https://api.aetherdb.ai/v1/documents?filename=invoice-8492.txt&content_type=text/plain" \
  -H "Authorization: Bearer $AETHER_API_KEY" \
  -H "Idempotency-Key: invoice-8492" \
  --data-binary @invoice-8492.txt

Without the header

For the async and batch routes — or for a key that must survive a document being updated — keep the mapping yourself:

1. Derive a stable key for the source in your own store.
2. Persist the doc_id Aether returns on first insert, keyed by that stable key.
3. On re-ingest, look the key up: no doc_id yet means insert; an existing doc_id means update.

update overwrites the document in place — same doc_id, fresh chunks and vectors — and replaces both the structured metadata map and the tag set. Resend the full sets you want the document to end up with, not just the entries that changed.

Tagging Documents

Tags are flat string labels you attach to a document so you can filter searches later — scope a query to one customer, one user, or one kind of content. Every insert variant accepts them (insert, insert_text, insert_stream, insert_async, insert_with_embeddings, and batch insert), and so does update. On the query side, search, retrieve, and search_by_vector all take a tags filter, and the semantics are AND: a document matches only if it carries every listed tag. See Tags and metadata for the search side, and Multi-tenant patterns for using tags to keep tenants' data apart.

Two rules for choosing tag values:

• No commas. Tags are sent comma-joined on the wire, so a comma inside a tag value will split it into two tags.
• Use key:value slugs. A consistent convention like customer:acme, user:42, kind:memory keeps filters readable and composable.

And one rule for updates: update replaces the document's tag set. Pass the full set of tags the document should end up with — tags from the original insert don't carry over.

# Tag a document at insert time
doc = client.insert_text(
    "Acme onboarding notes...",
    filename="notes.txt",
    tags=["customer:acme", "kind:memory"],
)

# update() replaces the tag set — pass every tag you want to keep
updated = client.update(
    doc.doc_id,
    "./data/notes-v2.txt",
    tags=["customer:acme", "kind:memory", "status:reviewed"],
)

Beyond the AND tags filter, search and list also accept OR facets — any_tags (match any listed tag), content_types, and sources — plus a source you can set on insert (a single origin label like slack, notion, or upload). Facets compose by AND across facets and OR within a facet. See Search filters.

One caveat on search filters: they're applied to a candidate set from vector retrieval, not during index traversal, so a narrow filter can return fewer than k results even when more matches exist. When you need the complete set of documents matching a metadata filter (not the top-k by similarity), use list with the same filters — it scans the full active set and paginates the entire matching population.

Listing and Downloading

The list method shows all active documents in your store — anything that hasn't been deleted. Because Aether collections can scale to millions of documents, the list method supports robust pagination using offset and limit.

Retrieving active datasets and downloading content.

# List active documents with pagination
active_docs = client.list(offset=0, limit=50)

for item in active_docs:
    print(item.doc_id, item.title)

# Download the raw payload of a document back to a file
bytes_written = client.download("doc_12345", output_path="./output.md")

Deletions

Removing a document in Aether does not immediately erase chunks on disk, but rather tombstones the metadata record via an event. Think of tombstoning like moving a file to the recycle bin — the document won't appear in search results or in list calls, but the underlying data is still there. This means deletions are safe and reversible. If you change your mind, you can call restore to bring a tombstoned document back into the active pool, and it will be searchable again immediately.

# Tombstone a document softly
client.delete("doc_12345")

# Restore a tombstoned document back to active pool
client.restore("doc_12345")

Permanent Deletion

Tombstoning is the right default, but sometimes recoverable isn't acceptable — a GDPR erasure request, or any compliance workflow that requires the data to be truly gone. For these cases the REST API supports hard deletion: pass hard=true on the DELETE request.

curl -X DELETE "https://api.aetherdb.ai/v1/documents/doc_12345?hard=true" \
  -H "Authorization: Bearer $AETHER_API_KEY"

Hard deletion takes effect immediately: vectors are removed from the search indexes, stored chunk data is deleted, and the document's encryption key is destroyed, so any archived copies become permanently unreadable (crypto-erasure).

This generic lifecycle applies to ordinary documents. Canonical conversation turns currently reject soft delete, hard delete, and restore with 409 thread_turn_immutable; there is no thread erasure endpoint yet.

The SDKs don't expose a hard-delete flag yet; delete in every SDK is tombstone-only. Call the REST endpoint directly when you need permanent removal.

Response shapes

Successful insert, update, batch-insert item, and BYOE insert responses return the same compact record:

{
  "doc_id": "doc_12345",
  "cid": "aether:mfrggzdfmztwq2lk...",
  "chunks": 1,
  "vectors": 1,
  "version": 1
}

GET /v1/documents/{id} expands that with metadata:

{
  "doc_id": "doc_12345",
  "cid": "aether:mfrggzdfmztwq2lk...",
  "title": "pto-policy.txt",
  "content_type": "text/plain",
  "size_bytes": 43,
  "chunks": 1,
  "vectors": 1,
  "version": 1,
  "created_at": "2026-06-25T18:30:00Z",
  "updated_at": null,
  "entity_id": null,
  "tags": ["customer:acme", "kind:memory"],
  "source": "upload"
}

tags and source are echoed on every document record (insert, fetch, and list) and on every search hit; source is null when the document had none.

Errors use the shared API error envelope:

{
  "error": "Document not found",
  "code": "not_found",
  "request_id": "req_..."
}

Common statuses are 400 for invalid input, 401 for missing or invalid authentication, 402 for plan limits, 404 for unknown or cross-tenant documents/jobs, 413 for request or batch size limits, 415 when binary content is sent to a synchronous route, 422 for unsupported content, 429 for rate limits or per-tenant job concurrency, and 500 / 503 for transient server errors. See Errors for retry guidance.