Fewer signs per sentence.

What this is, and what it isn't

The scope is deliberately narrow, and worth stating before any of the engineering.

Why autocomplete, and why a small model

Signing a full English sentence is expensive twice over: physically, and statistically.

If per-sign recognition accuracy is p, the chance an n-sign sentence survives end to end is roughly pn. The exponent is the problem:

A model of the failure mode, not a measurement — but it is why the architecture is shaped this way.

Why there's no trained classifier in the live path

A classifier's output layer is fixed to the vocabulary it was trained on. Adding one sign means collecting data, retraining, re-exporting and shipping a new model — incompatible with the thing this demo most wants to test: a user adding their own sign in a second and using it on the next frame. So detection for the demo is geometric:

21 MediaPipe landmarks
   → normalise around the wrist        (translation- and scale-invariant)
   → 72-dimension feature vector
   → RMS Euclidean distance to every template
   → nearest match, or UNKNOWN above τ = 0.42

sign-embeddings/ implements the fallback — reuse an internal layer of a trained TFLite model as an embedding extractor, match by cosine similarity — for when template matching runs out of headroom. Built and tested, not wired in, because the demo doesn't need it.

Why the LLM is small and rarely called

Glosses aren't English: no articles, no tense, no register. That's a language task, so a language model does it rather than the vision path. But the LLM is the slowest and hottest component, so the architecture minimises how often it runs — an exact sequence already confirmed returns from memory in about a microsecond, a sequence whose candidates the user has chosen from before returns from sentence memory, and only a genuinely new sequence reaches the 3B. That ordering is also what keeps the demo interactive under sustained camera load.

The pipeline

Every box runs on the phone. The only network traffic possible is a LAN socket to a second phone you pair yourself.

Recognition details

• Features. Landmarks normalised around the wrist, then expanded to 72 dimensions with the same transform the snapshot trainer used — so bundled and user-recorded templates are directly comparable.
• Capture gate. Landmark shape (no-hand), hand-to-camera distance (too-far), steadiness across the hold window (unstable) — surfaced as "Move your hand closer", "Hold your hand steady", rather than silently storing a bad template.
• Hold-to-confirm. A ring fills over ~1 s, with a cooldown that forces the hand to visibly re-form before the same gloss can fire again. On a call, a detection zone restricts matching to a hand held at sign height.
• User templates. App-private JSON, atomic temp-file-then-swap writes, a pub-sub so other mounted screens reload live, and a serialised save queue so concurrent captures can't corrupt the store.
• Demo limits. Single hand, static poses, one sign per hold. Two-handed and motion signs are not implemented.

Getting audio into a call

Android exposes no public API for injecting audio into a cellular call, which constrains the whole output stage. Two routes are implemented:

The autocomplete stores

Two stores, different jobs — one keyed on the gloss sequence, one on what the user actually picked.

Sentence memory — what the app ships

Records which candidate the user picked per exact gloss sequence and resurfaces it ahead of fresh generation, as app-private JSON with atomic writes. The Memory screen reviews and forgets entries. A sequence can legitimately mean different things on different occasions, so picking a different sentence later adds an option rather than replacing the earlier one — and once a sequence has enough remembered, the LLM isn't called for it again.

Translation memory — the engine behind it

A standalone retrieval engine implemented twice, Kotlin (Room/SQLite) and TypeScript, for corpora larger than a JSON file should hold. Dependency-free and independently tested.

• Exact path — O(1). Kotlin hashes the token characters in place (FNV-1a, no joined key string, zero allocation) into an open-addressing table; collisions resolve by verifying tokens, never by trusting the hash.
• Fuzzy path — two stages. An inverted-index walk accumulates IDF-weighted overlap into a reused, generation-stamped Float64Array; tokens in >60 % of memories are skipped unless the query has nothing else; unknown glosses get single-edit repair (METFORMIM → METFORMIN). The top 48 candidates are then scored 0.45·weightedJaccard + 0.25·queryCoverage + 0.30·lcsRatio — the LCS term makes sign order count. Ties break deterministically.
• Failure behaviour. Ids are assigned in RAM, so a confirmed phrase is queryable before SQLite has heard of it. A corrupt or full database flips the layer to memory-only and counts the failure; warm-up skips corrupt rows instead of failing start-up; malformed recogniser output is a returned value, never an exception.
• Bounded. 20 000 entries, least-used/least-recent eviction, pinned entries never evicted.

Models

One vision model, one language model, one speech model — and the component doing sign matching isn't a model at all.

No cloud inference: there is no OpenAI, Anthropic or Gemini key in the live app. GGUF choice per RAM tier (1.5B at 6 GB, 3B at 8 GB, 7B at 12 GB+) is documented in llm-testbed/, the harness used to benchmark candidates on the actual device.

On device — Snapdragon 8 Elite Gen 5

Status and limitations

Completed at the end of the hackathon. Demo-quality, and specific about which parts are which.

Known limitations

• Static single-hand poses only; recognition quality depends on lighting and framing.
• The capture gate is tuned by hand rather than from data.
• The relay assumes both phones share a network.
• All timing figures here are engineering benchmarks on a CI container or the demo device — none of the ergonomics have been evaluated with signers yet.

Next, in order

1. Ergonomic evaluation of the hold-and-confirm loop with signers.
2. Motion and two-handed signs.
3. Measured per-sign accuracy on the bundled set.
4. Wiring sign-embeddings if template matching saturates.

Run it yourself

A physical Android device is required — the camera path can't be demoed on an emulator.

git clone https://github.com/xreedev/mudra.git
cd mudra/app
npm install

# one-time: the LLM weights (Qwen2.5-3B-Instruct Q4_K_M, from Hugging Face)
adb shell mkdir -p /sdcard/Android/data/com.mudraapp/files/models
adb push qwen2.5-3b-instruct-q4_k_m.gguf /sdcard/Android/data/com.mudraapp/files/models/

npm start          # Metro
npm run android    # build + install

For a release APK the model is bundled instead of pushed — drop it into android/app/src/main/assets/models/ and the app extracts it on first launch. On Windows, scripts/build-release-with-llm.ps1 automates that, plus the MAX_PATH and asset-compression workarounds a multi-GB asset needs.

Then try the loop

1. Add custom sign — hold a sign inside the guide circle until the ring completes, name it, save. Usable on the next frame.
2. Call — hold signs to build a gloss sequence, pick one of the proposed sentences, it's spoken.
3. Memory — the sentence you picked comes back next time those signs do.

Two-phone relay, optional

Both phones on the same Wi-Fi: one opens Call, the other Receive. Discovery is zeroconf/mDNS (aslrelay), the session runs over TCP on port 12345. Guest networks with client isolation block it — a phone hotspot is the reliable fallback. Without a relay it still works on one phone, over speakerphone.

The standalone modules

Each builds and tests on a laptop — no emulator, no device, no network:

cd memory-layer     && ./gradlew :core:test   # Kotlin engine, no Android SDK needed
cd memory-layer-rn  && npm install && npm test
cd sign-embeddings  && npm install && npm test

Full setup, troubleshooting and the technical detail behind every claim on this page is in the README.