Multimodal memory: images and audio

Remember screenshots, photos, diagrams, recordings, and voice notes with the same entity-scoped Memory API you use for text. Aether retains the original media, indexes a caption or transcript, and returns text, image, and audio results in one relevance-ranked recall.

How it works

1. Your SDK loads an image or audio source and sends its bytes to Aether over the authenticated API.
2. You can supply a caption/transcript, or let an operator-configured VLM/transcription route derive one.
3. Aether stores the original through its encrypted document path and embeds the bounded derived text in the normal text vector space.
4. recall blends all modalities. Each media result carries modality: "image" | "audio" so your app can render it appropriately.

This design uses one ranking space instead of a separate CLIP index: image captions, audio transcripts, and text memories are directly comparable, with the same tenant, partition, metadata, and read-ACL behavior.

Python

from aether import Memory

memory = Memory("user-42")  # reads AETHER_API_KEY

# bytes, a local path, an HTTP(S) URL, a file-like object, or PIL.Image
image = memory.remember(
    image="https://example.com/architecture.png",
    metadata={"project": "launch"},
)

# Automatic transcription. Pass transcript="..." to supply your own instead.
audio = memory.remember(audio="meeting.m4a", transcribe=True)

for hit in memory.recall("what are the launch milestones?"):
    print(hit.modality or "text", hit.text, hit.id)

Python also exposes the identical calls on AsyncMemory with await.

TypeScript

import { Memory } from "@aether-ai/sdk";

const memory = new Memory("user-42");

// Uint8Array, ArrayBuffer, Blob, URL, HTTP(S) string, or Node file path
const image = await memory.remember({
  image: new URL("https://example.com/architecture.png"),
  metadata: { project: "launch" },
});

const audio = await memory.remember({
  audio: "./meeting.m4a",
  transcribe: true,
});

for (const hit of await memory.recall("what are the launch milestones?")) {
  console.log(hit.modality ?? "text", hit.text, hit.id);
}

.NET

using Aether.Sdk;

using var memory = new Memory("user-42");

var image = await memory.RememberImageFileAsync(
    "architecture.png",
    metadata: new Dictionary<string, object?> { ["project"] = "launch" });

var audio = await memory.RememberAudioFileAsync("meeting.m4a", transcribe: true);

foreach (var hit in await memory.RecallAsync("what are the launch milestones?"))
    Console.WriteLine($"{hit.Modality ?? "text"} {hit.Text} {hit.Id}");

Use RememberImageAsync(byte[]) or RememberAudioAsync(byte[]) when your app already owns the bytes.

Go

package main

import (
	"context"
	"fmt"
	"log"

	aether "github.com/quintessence-group/aether-sdk-go"
)

func main() {
	ctx := context.Background()
	memory, err := aether.NewMemory("user-42")
	if err != nil { log.Fatal(err) }

	// []byte, local path, HTTP(S) URL string, or *url.URL
	_, err = memory.RememberImage(ctx, "https://example.com/architecture.png",
		aether.WithMediaMetadata(aether.Metadata{"project": "launch"}))
	if err != nil { log.Fatal(err) }

	_, err = memory.RememberAudio(ctx, "meeting.m4a")
	if err != nil { log.Fatal(err) }

	hits, err := memory.Recall(ctx, "what are the launch milestones?")
	if err != nil { log.Fatal(err) }
	for _, hit := range hits {
		modality := "text"
		if hit.Modality != nil { modality = *hit.Modality }
		fmt.Println(modality, hit.Text, hit.ID)
	}
}

Supplying your own caption or transcript

Caller-derived text avoids automatic model egress and makes ingestion deterministic:

memory.remember(image=image_bytes, caption="A whiteboard with Q3 release milestones")
memory.remember(audio="meeting.wav", transcript="The team agreed to ship Tuesday.")

The other SDKs expose caption/transcript in their media options. Setting audio transcribe=false without an explicit transcript is a client-side error.

Storage, playback, and security

• The original media follows the normal document path: envelope encryption, tenant/partition isolation, optional read ACL, and the deployment's configured storage backend.
• Recall returns the indexed caption/transcript, not binary bytes. Use the result's document id with the raw client's download method when your authorized app needs to display or play the original.
• MIME declarations are checked against file signatures. Supported images are JPEG, PNG, WebP, and GIF; supported audio is MP3, MP4/M4A, WAV, WebM, Ogg, and FLAC.
• Automatic caption/transcription is disabled unless the node operator configures the corresponding provider route and credential. Provider errors surface normally; Aether never accepts a provider URL from the media request.
• A strict PHI-egress tenant requires a separate operator-designated media route and never falls back to the normal one. That routing control is not, by itself, a BAA or compliance certification.

Mixed recall

Every hit keeps the familiar id, text, metadata, and score fields. Media adds one rendering hint:

modality = absent   → text memory
modality = image    → text is the indexed image caption
modality = audio    → text is the indexed audio transcript

The original remains addressable by id; search and list responses never inline it. See the Memory API reference for the wire contract and Search & Retrieval for ranking/filter behavior.