Documents API

Manage and query documents within the Aether Object Store using the Aether SDK.

Documents are the foundation of Aether. Before you can search for anything, you need to store your content — PDFs, text files, markdown, or raw strings. When you insert a document, Aether automatically splits it into chunks, generates vector embeddings for each chunk, and indexes everything for search. You don't need to worry about the chunking or embedding process — it happens behind the scenes.

SDK methods

Every document operation is a method on the AetherClient. Names follow each language's conventions; full usage for each is shown in the examples below this table.

¹ The .NET SDK has no text helper — decode the bytes from DownloadAsync, e.g. Encoding.UTF8.GetString(bytes).

Your API key is set once when you construct the client (see Authentication); the SDK attaches it to every request automatically. Failed calls raise an SDK error that maps to the status codes documented in Errors.

Document route contract

The SDK methods above wrap the hosted REST contract. Use the SDK for application code; use this route table when you are debugging, writing a proxy, or checking the wire shape.

Direct REST requests authenticate with Authorization: Bearer <api-key>. SDK users should not set this header manually — construct the client with an API key and let the SDK attach it.

Chunking and CIDs

Aether stores documents as content-addressed chunks:

• The default storage chunk size is 256KB.
• Insert and update calls can pass chunk_size; SDKs expose it as chunk_size, chunking.chunkSize, ChunkingConfig.ChunkSize, or aether.WithChunkSize(...) depending on language.
• overlap is optional and defaults to 0 when a custom chunk_size is used.
• Synchronous HTTP requests are capped at 50MB by the API layer. The node also enforces its configured max_document_bytes limit before work is accepted. Use async ingestion for parser-backed formats and long-running uploads.
• Each stored chunk gets a deterministic CID built from a BLAKE3 hash of the chunk bytes and encoded with the aether: prefix.

The top-level document cid in a DocumentRecord is stable for identical content. Chunk CIDs let Aether verify and route stored bytes across swappable storage backends without depending on filenames or database row ids.

Supported file types

Text and office formats are extracted and indexed automatically. Sending an unsupported format returns 422.

Interacting with Documents

The AetherClient makes accessing documents simple.

Inserting Documents

There are two ways to add content. insert_text is the quickest way to get started — just pass a string directly and Aether takes care of the rest. Use insert when you have a file path or binary data (like a PDF or an uploaded file buffer). Both methods return a DocumentRecord that tells you the assigned doc_id, how many chunks the content was split into, and how many vectors were generated. This is useful for verifying that your document was processed correctly — if a document has zero chunks, something likely went wrong with the input.

# Insert from a path
doc = client.insert(
    file_path="./data/manifesto.md", 
    content_type="text/markdown"
)

# Insert raw text string directly
doc_text = client.insert_text(
    text="Some text in the object store.",
    filename="text.txt"
)

print(doc_text.doc_id, doc_text.chunks)

Both methods return a DocumentRecord containing the doc_id, cid (Content Identifier), and metadata regarding the number of chunks and vectors generated.

Quotas & Plan Limits

As a fully managed service, Aether enforces storage and throughput bounds based on your current organization's subscription tier. If an insertion or embedding request exceeds your permitted quota (e.g., maximum storage bytes, or maximum active documents), the API will reject the request with a 402 Plan limit exceeded status. Your applications should be prepared to handle these 402 responses or monitor your usage periodically via the Dashboard to avoid unexpected ingestion interruptions.

Streaming Large Files

When inserting multi-gigabyte files, loading the entire payload into RAM before transmission can lead to memory exhaustion. To prevent this, the SDKs provide an insert_stream method that streams the binary data directly to Aether's load balancers.

# Pass a file object opened in binary mode
with open("./data/massive-knowledge-base.zip", "rb") as f:
    doc = client.insert_stream(
        stream=f,
        filename="massive-knowledge-base.zip",
        content_type="application/zip"
    )
print(doc.doc_id)

Async Ingestion

For large files or batch uploads, use async ingestion to avoid blocking. insert_async returns immediately with a job handle (job_id, status, poll_url); wait_for_job then polls until the job reaches a terminal state — completed or failed — and returns the final status (doc_id on success, error on failure). It raises a timeout error if the job doesn't finish within the deadline (60 s by default).

# Start an async insert — returns immediately with {job_id, status, poll_url}
job = client.insert_async(file_path="./data/large-report.pdf", content_type="application/pdf")
print(f"Job started: {job['job_id']}")

# Block until the job reaches a terminal state (polls in the background)
result = client.wait_for_job(job["job_id"])

if result["status"] == "completed":
    print(f"Document ready: {result['doc_id']}")
else:
    print(f"Job failed: {result.get('error')}")

The raw job endpoint returns a progress snapshot:

{
  "job_id": "8d3f1c26-9ea1-4ec9-b701-6e3f1c2a7a01",
  "status": "embedding",
  "doc_id": null,
  "message": "Embedding document",
  "progress": 0.6,
  "created_at": "2026-06-25T18:30:00Z",
  "finished_at": null,
  "error": null
}

Terminal states are completed and failed. Jobs are tenant-scoped; polling a job from another tenant returns 404 rather than leaking that it exists.

Custom Embeddings (BYOE)

If you use your own embedding model, you can insert documents with pre-computed vectors instead of relying on Aether's built-in embedder. Provide either a single embedding for the whole document, or per-passage embeddings for fine-grained retrieval.

# Insert with per-passage embeddings
doc = client.insert_with_embeddings(
    content="Full document text...",
    filename="report.md",
    passages=[
        {"text": "First paragraph...", "embedding": [0.1, 0.2, ...]},
        {"text": "Second paragraph...", "embedding": [0.3, 0.4, ...]},
    ]
)

Updating and Retrieving Metadata

Sometimes you need to replace a document's content — maybe you fixed a typo, or a newer version of the file is available. Calling update replaces the content entirely: Aether re-chunks the new content, generates fresh embeddings, and re-indexes everything. The old vectors are discarded. On the other hand, get is a lightweight call that just returns metadata (like chunk count, file size, and content type) without downloading the actual content. It's a quick way to check on a document without pulling the full payload. Document records also carry timestamps: get returns both created_at and updated_at, and each item from list includes created_at.

You can update the actual contents of an existing document or fetch its metadata details.

# Overwrite an existing document by doc_id
updated = client.update("doc_12345", "./data/revised_manifesto.md")

# Simply fetch the latest metadata about the document
meta = client.get("doc_12345")
print(f"Chunks: {meta.chunks}, Size: {meta.size_bytes}")

Tagging Documents

Tags are flat string labels you attach to a document so you can filter searches later — scope a query to one customer, one user, or one kind of content. Every insert variant accepts them (insert, insert_text, insert_stream, insert_async, insert_with_embeddings, and batch insert), and so does update. On the query side, search, retrieve, and search_by_vector all take a tags filter, and the semantics are AND: a document matches only if it carries every listed tag. See Filtering by tags for the search side, and Multi-tenant patterns for using tags to keep tenants' data apart.

Two rules for choosing tag values:

• No commas. Tags are sent comma-joined on the wire, so a comma inside a tag value will split it into two tags.
• Use key:value slugs. A consistent convention like customer:acme, user:42, kind:memory keeps filters readable and composable.

And one rule for updates: update replaces the document's tag set. Pass the full set of tags the document should end up with — tags from the original insert don't carry over.

# Tag a document at insert time
doc = client.insert_text(
    "Acme onboarding notes...",
    filename="notes.txt",
    tags=["customer:acme", "kind:memory"],
)

# update() replaces the tag set — pass every tag you want to keep
updated = client.update(
    doc.doc_id,
    "./data/notes-v2.txt",
    tags=["customer:acme", "kind:memory", "status:reviewed"],
)

One caveat on filtered searches: tag filters are applied to a candidate set retrieved from the whole store, not during index traversal. When a tag matches only a small slice of your documents, a filtered search can return fewer than k results even though more matches exist. If you're filtering to a narrow tag, request a larger k than you need, then keep only results above your score threshold.

Listing and Downloading

The list method shows all active documents in your store — anything that hasn't been deleted. Because Aether collections can scale to millions of documents, the list method supports robust pagination using offset and limit.

Retrieving active datasets and downloading content.

# List active documents with pagination
active_docs = client.list(offset=0, limit=50)

for item in active_docs:
    print(item.doc_id, item.title)

# Download the raw payload of a document back to a file
bytes_written = client.download("doc_12345", output_path="./output.md")

Deletions

Removing a document in Aether does not immediately erase chunks on disk, but rather tombstones the metadata record via an event. Think of tombstoning like moving a file to the recycle bin — the document won't appear in search results or in list calls, but the underlying data is still there. This means deletions are safe and reversible. If you change your mind, you can call restore to bring a tombstoned document back into the active pool, and it will be searchable again immediately.

# Tombstone a document softly
client.delete("doc_12345")

# Restore a tombstoned document back to active pool
client.restore("doc_12345")

Permanent Deletion

Tombstoning is the right default, but sometimes recoverable isn't acceptable — a GDPR erasure request, or any compliance workflow that requires the data to be truly gone. For these cases the REST API supports hard deletion: pass hard=true on the DELETE request.

curl -X DELETE "https://api.aetherdb.ai/documents/doc_12345?hard=true" \
  -H "Authorization: Bearer $AETHER_API_KEY"

Hard deletion takes effect immediately: vectors are removed from the search indexes, stored chunk data is deleted, and the document's encryption key is destroyed, so any archived copies become permanently unreadable (crypto-erasure).

The SDKs don't expose a hard-delete flag yet; delete in every SDK is tombstone-only. Call the REST endpoint directly when you need permanent removal.

Response shapes

Successful insert, update, batch-insert item, and BYOE insert responses return the same compact record:

{
  "doc_id": "doc_12345",
  "cid": "aether:mfrggzdfmztwq2lk...",
  "chunks": 1,
  "vectors": 1,
  "version": 1
}

GET /documents/{id} expands that with metadata:

{
  "doc_id": "doc_12345",
  "cid": "aether:mfrggzdfmztwq2lk...",
  "title": "pto-policy.txt",
  "content_type": "text/plain",
  "size_bytes": 43,
  "chunks": 1,
  "vectors": 1,
  "version": 1,
  "created_at": "2026-06-25T18:30:00Z",
  "updated_at": null
}

Errors use the shared API error envelope:

{
  "error": "Document not found",
  "code": "not_found",
  "request_id": "req_..."
}

Common statuses are 400 for invalid input, 401 for missing or invalid authentication, 402 for plan limits, 404 for unknown or cross-tenant documents/jobs, 413 for request or batch size limits, 415 when binary content is sent to a synchronous route, 422 for unsupported content, 429 for rate limits or per-tenant job concurrency, and 500 / 503 for transient server errors. See Errors for retry guidance.