CactusBrain developer guide

Platform overview

CactusBrain is four on-device SDKs that share one project key and one model-delivery pipeline. Pick the ones your app needs:

• Vision — detect objects and read text (OCR), and recognize your own objects from example images you enroll.
• Text — embeddings, ranking and small-model chat, all local.
• Personalize — rank items to one user's behaviour on their own device. No model, no server.
• Signal — tag what a sound is, and enhance speech, from sampled audio.

Every SDK includes the native runtime it needs. The SDKs give your app task-based APIs so you never load a model or prepare tensors yourself; CactusBrain (the workspace) manages projects, access and delivery.

Quick start

Vision, Text and Signal share three steps:

1. Configure a session with your project key.
2. Prepare a model — the SDK downloads, verifies and installs it once, with progress you can show.
3. Run — detect, read, embed, chat, rank or tag, all on the device.

Full, copy-pasteable walkthroughs live in the per-platform guides:

• Vision for iOS
• Vision for Android
• Text (embeddings, ranking, chat)
• Signal (audio tagging)
• Personalize (local ranking, no model download)

The rest of this page is the concept reference the guides link back to: what each capability does, and how model delivery, entitlement and billing work underneath.

Create an account

Create a developer account to use the workspace. You can organize projects and see the models and SDK files assigned to your organization. An account alone does not grant access to a model.

Vision overview

Vision does two kinds of work, both on the device:

• Detection and OCR — find known objects in an image and read text out of it, using a delivered model. No setup beyond preparing the model.
• Identification — recognize your objects. You enroll reference images into a collection, then ask which enrolled object a new image matches.

The two are independent: an app can use detection and OCR without ever creating a collection, and identification is available whenever a compatible embedding model is prepared. For installation and runnable code, follow the iOS or Android guide; the sections below describe the identification API in detail.

Detection and OCR

Once a model is prepared, detect returns the objects it found — each with a label, a confidence and a bounding box — and ocr returns the text in the image. Two OCR models are available: a small fast English model and a larger multilingual one.

Both take encoded image bytes, so your app controls the conversion. See the iOS and Android guides for the exact calls, including the EXIF-orientation detail that keeps returned coordinates lined up with the image the user sees.

Face primitives

Faces are four models, downloaded separately and run separately. There is no bundled “face verification” feature, and that is deliberate: an app that blurs faces in a photo should not have to ship a spoof detector, and shipping them as one product would make a claim about what your app does that only you can make.

• Detect — where the faces are. Two checkpoints: one for a face filling the frame, one for smaller faces further away. They differ in input resolution, not in quality.
• Liveness — whether a detected face is a person or a picture of one. Returns three probabilities — live, print attack, replay attack — and no verdict.
• Embed — turn one detected face into a vector.
• Compare — a cosine similarity between two vectors from the same model.

Detection runs first; the other three take a face box as their input. Comparison is pairwise and stateless: there is no gallery, no index and no search, and the SDK stores no vector. An embedding is biometric data, and where it may be kept is a decision your jurisdiction makes rather than one we should make convenient by accident.

The Vision overview shows measured output from each of the four, including which of the two comparison models can actually tell two people apart.

Collections

A collection stores the objects you want to recognize. Its ID becomes the saved file name. IDs can use letters, numbers, -, _, and ., up to 128 characters.

The model package and SDK decide how matching works. Before it starts matching, CactusBrain checks that an existing collection still works with the current model and vector size.

Enrollment

Add one object with one or more reference images. Each image gets its own vector. Adding an existing object ID replaces its images and details together.

try await collection.add(
    id: "coke-500",
    images: referenceImages,
    metadata: ["displayName": "Coke 500 ml"]
)

let count = await collection.identityCount()

Use different, clear images of the same object. You need at least one image. The released model will define the quality guidelines.

Identification

Identification turns the query image into a vector and compares it with your saved images. It returns up to topK matches when the best match passes the collection's rules.

let result = try await collection.identify(
    image: queryImage,
    topK: 3
)

if result.identified, let match = result.matches.first {
    print(match.id, match.score, match.metadata)
}

The result includes the best score and the total SDK time in milliseconds. Ties are always sorted by object ID.

Unknown handling

When nothing matches, identified is false and matches is empty. Check reason to tell the difference between no good match and an empty collection.

switch result.reason {
case .noMatch:
    print("No enrolled identity matched")
case .emptyCollection:
    print("Enroll at least one identity before identifying")
case nil:
    break
}

Offline and privacy

Your app needs a network connection once, to get permission and download a model the first time. After the model is installed and checked, matching and inference run on the device with no network connection.

Reference images, query images, vectors and results stay inside your app. CactusBrain does not offer cloud inference and never receives this content.

Model compatibility

The workspace shows the model packages available to your organization and projects. You can use a model only when the workspace shows that your project has access and your platform is supported.

Collections save the modelIdentifier and embeddingDimensions. CactusBrain stops if you open a collection with a different model or vector size.

Text overview

CactusBrain Text runs three things on the device from the same package and project key as Vision: embeddings (text to vectors for search, clustering and deduplication), ranking (score documents against a query, optionally personalized), and chat (streamed replies from a small local model). Prompts, replies and embedded text never leave the device.

Chat models are several hundred megabytes, so prepare them where you can show progress rather than inside the first message. The full API, both platforms and the honest limits of small local models are in the CactusBrain Text guide.

Overview

CactusBrainPersonalize ranks the items and vectors your app provides. It runs locally. It does not run a model, call a service, upload data, or need a server. Your app chooses the items, creates the vectors, records events, and manages sessions.

Each user has a long-term preference vector. CactusBrain ranks candidates by how close their vectors are to it. Higher scores come first. Ties use item ID. Existing observe(user:item:event:) and rank(user:candidates:limit:) calls keep this behavior.

Temporal profiles

Long-term preference keeps all past activity. Recent interest and optional app sessions fade over time. The default recentHalfLife is seven days.

try await personalize.observe(
    user: "local-user",
    item: "article-123",
    event: .opened,
    timestamp: Date(),
    session: "morning-read"
)

When a new event arrives, older events count less. Positive and negative events fade at the same rate. A recent dismissal can change short-term recommendations without changing the long-term profile.

• Updates grow with vector size.
• Ranking still checks the candidates you provide.
• CactusBrain does not keep a full history of events.
• Use session IDs only for sessions you want to keep active.

Ranking modes

.longTerm is the default. It uses only the long-term profile. .hybrid(session:) combines long-term, recent, and session scores when they exist. Its default weights are 0.5, 0.3, and 0.2.

let results = try await personalize.rankDetailed(
    user: "local-user",
    candidates: candidateIDs,
    limit: 10,
    mode: .hybrid(session: "morning-read"),
    at: Date()
)

try await personalize.clearSession(
    user: "local-user",
    session: "morning-read"
)

Use rankDetailed when you need the total score and each part of it. The total score decides the order.

Persistence

Personalize saves items, settings, and user profile totals in a local JSON file, and appends each new interaction to a second file next to it. The interaction file is folded back into the first when it grows past its limit, or whenever the catalogue changes, so recording a tap does not rewrite the catalogue. Older version 1 files keep their long-term profile on upgrade; recent and session data start empty because those files did not save them. A store opened at version 3 cannot be reopened by an older SDK.

Use profile(for:) to see safe summary details, such as event count, last event, and active session IDs. Call clearSession(user:session:) when your app session ends. It only removes that session.

Signal overview

CactusBrain Signal runs models over sampled audio on the device. It does two jobs: tagging (label what a clip sounds like) and enhancement (clean up speech). Both go through the same delivery pipeline as Vision and Text, so you configure a session with a project key and never name a model file, URL or key.

Configure through CactusBrainSignalSDK.configure(projectKey:) to get a SignalSession. The tagging model loads on first use, so an app with a microphone button that is never pressed pays nothing for it. The session also reports the captureFormat the installed model needs, so you sample audio at the rate it expects rather than guessing.

For installation and runnable code on both platforms, follow the CactusBrain Signal guide; the sections below describe the shared audio-input contract.

Tagging

session.tags(for:) takes an audio input and returns labels, each with a score. Call prepare() first when you have somewhere to show progress; otherwise the first tags call downloads and loads the model inline. unload() releases the loaded model while leaving the installed file on disk.

Enhancement rides the same signal capability — both are “run a model over sampled audio” — and exposes streaming enhance(chunk:) for live capture.

Signal input contract

let input = SignalInput.pcm(
    samples: interleavedSamples,
    sampleRate: 44_100,
    channels: 1
)

try await recommender.add(
    .init(id: "track-001", signal: input)
)

SignalInput.file needs an existing local file URL. SignalInput.pcm needs nonempty, valid floating-point samples with a positive sample rate and channel count. Bad PCM is rejected before the model runs.

Signal personalization

CactusBrainSignalPersonalizer turns a signal into a vector and hands it to Personalize as an ordinary item. Once the vector exists, long-term, recent, session and hybrid ranking all work exactly as they do for text and images — a sound becomes just another thing a user can prefer.

Capability registry

The Swift CactusBrain product tells your app whether each feature is ready: vision.identify, vision.ocr, text.embedding, signal.tagging and personalize.ranking. It reports product status, not low-level runtime details.

let signal = await CactusBrain.capability(.signalTagging)

switch signal.state {
case .available:
    print(signal.provider, signal.model?.identifier)
case .modelRequired, .notInstalled, .runtimeUnavailable, .unsupportedDevice:
    break
}

Personalize ranking works without a model. Vision, Text and Signal each need a compatible model prepared before their capabilities report ready.

Model catalog resolution

CactusBrainCapabilityRegistry checks whether a model can run, whether this exact model is supported, whether the device qualifies, and whether the model is installed. A model entry includes its ID, name, version, size, supported features, and device needs.

let nemotron = CactusBrainModelDescriptor(
    id: "nvidia/nemotron-ocr",
    name: "Nemotron OCR",
    capabilities: [.visionOCR],
    requiredRuntimeCapabilities: [.visionTextRecognition]
)

let registry = CactusBrainCapabilityRegistry(
    models: [nemotron],
    runtimeSupport: .init(
        supportedRuntimeCapabilities: [.visionTextRecognition],
        modelCompatibility: [nemotron.id: .unverified]
    ),
    device: .init(platform: .iOS, operatingSystem: .init(major: 17))
)

These are separate checks. vision_text_recognition means the native runtime has OCR code; it does not mean it can load a particular OCR checkpoint. An unverified model stays modelRequired. Only modelSupported lets it move to not-installed, then available. When more than one model fits, CactusBrain always chooses the same one by model ID.

This registry only reports status. It does not download, verify, load, update or run a model.

Model delivery

The CactusBrain SDK downloads a model when your app needs it. You assign an approved model package to a project; the SDK checks access, downloads the encrypted file, verifies it, stores it and loads it. Your users never manage model files.

Open Models to see model access and versions. Open a project’s Activity page to inspect its delivery requests. A model in the public catalog is not automatically ready for your project.

Model format

Use model packages built for the CactusBrain runtime. A package keeps the weights, tokenizer or input files, model details and compatibility details together. Do not use an upstream checkpoint unless it was converted and tested for this SDK release. Each package declares:

• Model identifier and task
• Required runtime version
• Input and output contract
• Supported platforms and backends
• License and commercial-use terms

Runtime

The native runtime is bundled with every CactusBrain SDK. Each SDK handles model delivery and local execution, so your app never loads a model or prepares tensors itself.

• Models run inside your app.
• Your inputs are not sent to a CactusBrain inference API.
• Model and backend support is checked for each release, so an artifact your runtime cannot load is refused before the download starts.

Billing

CactusBrain bills private-model storage, encrypted model delivery and operational features. It does not bill tokens, prompts, inference requests, CPU time, GPU time or monthly active users. Inference stays on the device.

Delivery usage is the byte count served by the authorized artifact response. Full responses count the encrypted artifact size; resumed range responses count only their returned range. Resolve requests, telemetry events, installations, local model loads and inference have no billing meter. Direct-origin accounting records bytes CactusBrain served, not a guarantee that a client finished installing.

A Builder app pays $0 for inference and $0 for active users however many people use it, because inference runs on the device and is never metered. What it pays for is delivery. Builder includes 300 GB a month and keeps serving to a ceiling of 375 GB, which is 125% of the included allowance. Delivery past the included allowance is not separately invoiced today; the ceiling is a cap rather than a meter. At the ceiling a resolve returns 429 and model delivery pauses until the period resets, so usage cannot run up an unbounded invoice. With an 80 MB model that ceiling is roughly 5,000 downloads a month, so size a plan on delivery bytes rather than on user count. Explorer has no headroom: its delivery stops at the included 50 GB.