Guides

Fact extraction

Turn raw text into atomic, retrievable facts at insert time. Instead of storing a 50 KB transcript and re-reading it on every query, Aether can distill it into a handful of standalone facts — "John fears flying", "John lives in Denver" — each stored as its own searchable memory.

This is the difference between a search index over your documents and a memory that learns. With extract_facts, one remember call about a long conversation becomes a set of clean facts your agent can recall cheaply and precisely, instead of dragging back the whole transcript.

How it works

When you insert with extract_facts=true, the server:

  1. Stores your document normally (the raw text is always kept).
  2. Calls an LLM (Anthropic Claude Haiku by default) to distill the text into a list of atomic facts.
  3. Stores each fact as its own sibling document, with:
    • a kind:fact tag, so you can retrieve facts only;
    • a parent:<source_doc_id> tag linking it back to the source;
    • the source's entity_id, partition, source label, and tags, all inherited.

Facts are ordinary documents — embedded, searchable, and filterable like anything else. The raw document stays too, so nothing is lost; you choose at query time whether you want facts, raw documents, or both.

Opt-in, and configured per node

extract_facts is off by default — existing inserts are unchanged. It requires an extraction LLM to be configured on the node (an API key plus an optional model override). If it isn't configured, an extract_facts=true request fails with 400 so the behavior is never silently dropped.

Extracting on insert

Set extract_facts when you insert text. The insert response reports how many facts were stored in facts_extracted.

curl -X POST "https://api.aetherdb.ai/documents?filename=session.txt&content_type=text/plain&entity_id=patient-john&extract_facts=true" \
  -H "Authorization: Bearer $AETHER_API_KEY" \
  --data-binary "John said he is scared of planes. He moved to Denver last year."
# → { "doc_id": "...", "facts_extracted": 2, ... }

remember still makes a single call and returns the raw memory; the fan-out into facts happens server-side. The raw client exposes the same flag directly: client.insert_text(text, extract_facts=True) (Go: aether.WithExtractFacts(true)).

Retrieving facts

Facts are searchable like any document. Pass kind=fact to a search or list to get only facts; omit it to get both facts and raw documents.

# Only the distilled facts for this patient
curl "https://api.aetherdb.ai/search?q=travel%20anxiety&k=5&kind=fact&entity_id=patient-john" \
  -H "Authorization: Bearer $AETHER_API_KEY"

# Or list every fact for the entity
curl "https://api.aetherdb.ai/documents?kind=fact&entity_id=patient-john" \
  -H "Authorization: Bearer $AETHER_API_KEY"

Because each fact inherits the source's entity_id and partition, the usual entity, time, and partition filters compose with kind=fact exactly as they do for raw documents.

Consolidation: facts that learn over time

Extraction alone would store a new fact every time the same thing comes up — "John fears flying" from Monday's session, "John gets anxious on planes" from Friday's. After a few weeks the fact list is full of near-duplicates and retrieval gets noisier.

So Aether consolidates facts. After extraction, each new fact is matched against the facts already known for that entity:

  1. Similarity check. The new fact is compared (vector similarity) to existing facts in the same entity_id scope. Only close matches advance — so consolidation is cheap.
  2. Merge decision. For a close match, an LLM decides whether the two are really the same fact. If yes, it produces one merged sentence that keeps any added detail.
  3. Fold in. The existing fact is updated with the merged text; its provenance grows to include the new source, its confidence increments (it's now corroborated by more sessions), and its last-seen time is refreshed. The redundant new fact is removed.

The result: one entry per distinct fact, each carrying how many sources back it and when it was last seen — the signal that turns a flat fact list into a memory that gets sharper over time. Consolidation runs in the background, so remember(extract=True) stays a single fast call.

Worked example

StepStored facts for patient-john
Session 1: "John is scared of planes." → extract"John fears flying." — confidence 1, sources: [s1]
Session 2: "Anything with turbulence makes John panic." → extractnew fact arrives: "John panics in turbulence."
…consolidation runssimilarity match → LLM says "same fact" → merge
Result"John fears flying, especially turbulence." — confidence 2, sources: [s1, s2], last-seen: session 2

A session that asserts something genuinely new ("John started flying again after therapy") stays a separate fact — consolidation only merges what the model confirms is the same.

Listing consolidated facts

list_extracted_facts returns an entity's consolidated facts, highest-confidence first — the clean, high-signal view of what's known.

memory = Memory(entity_id="patient-john")
for fact in memory.list_extracted_facts():
    print(fact.text)   # most-corroborated facts first

Two kinds of "facts"

list_extracted_facts returns the free-text facts distilled from your documents (this guide). It is distinct from the memory graph's list_facts, which returns structured predicate = value triples you write explicitly. Different tools for different jobs — extracted facts capture "what was said", graph facts model "what is true" as typed data.

Confidence pairs naturally with recency: rank by recency to surface what's current, and lean on confidence to trust what's corroborated across many sessions.

Backfilling existing documents

Already have documents stored without extraction? Run the backfill endpoint to distill facts from one retroactively. It reads the stored document, extracts facts, and stores them as siblings just like an insert-time extraction.

curlbash
curl -X POST "https://api.aetherdb.ai/documents/{id}/extract" \
  -H "Authorization: Bearer $AETHER_API_KEY"
# → { "doc_id": "...", "facts_extracted": 3, "fact_doc_ids": ["...", "...", "..."] }

The endpoint is tenant-scoped: a document id you don't own returns 404.

Cost & configuration

  • Model. Anthropic Claude Haiku by default (fast and inexpensive). The node operator can point extraction at a different model or a compatible endpoint.
  • Billing. Extraction LLM tokens are metered to your account alongside your other usage. Extraction is opt-in precisely so you only pay for it where the distilled-fact tradeoff is worth it.
  • What gets extracted. The model is prompted to keep only durable, self-contained facts and to drop questions, greetings, and hypotheticals. The extraction prompt is versioned in the engine so its behavior changes are explicit.

Facts complement, not replace

The raw document is always stored. Use raw documents when you need the full context or exact wording, and facts when you want cheap, high-signal recall. A common pattern: remember(extract=True) on every conversation turn, then recall(..., kind=fact)-style retrieval for the agent's working memory.

Next steps