Centralized ASR Model Management

A single system for the AI team to upload, configure, deploy, and track ASR models — replacing the scattered mount + git-configs + manual DB-script process we have today.

1The problem today

Getting a model live currently takes three separate hand-offs, and the "truth" about what's running lives in three different places:

• The model files sit on a mount.
• The config sits in a GitHub repo (bodhi_asr_configs) with many branches.
• What's actually running lives in a Postgres table that a devops script writes.

Nobody can look in one place and answer "which model is running where, and for whom?" New people have nothing to read. It works only because a few people remember how.

The fix: one system that is the single source of truth — one place to upload, configure, deploy, and see everything.

2The big picture

One catalog (a smart list of every model) plus one dashboard to manage it. Everything the AI team does happens on the dashboard; everything else (encrypt, store, deploy, restart) happens automatically behind it.

  AI team                    Catalog (the single source of truth)              Servers
 ┌────────┐   upload/config  ┌──────────────────────────────────┐   read at    ┌──────────┐
 │Dashboard├─────────────────►│  model list, configs, who-can-use,├─────boot────►│ bodhi-go │
 │ or CLI │                  │  what's live in dev vs prod       │              │ (dev)    │
 └────────┘   click "Deploy" └──────────────────────────────────┘              ├──────────┤
      │                                    │                                    │ bodhi-go │
      │                                    ▼                                    │ (prod)   │
      │                    ModelKits (encrypted, in OCI registry)               └──────────┘
      └──────► Jenkins pipeline ─► restarts the right servers ─────────────────────┘

3Architecture: one platform DB, two environments

Everything runs on the new API Platform databases. The old streaming-asr DB is being retired, so both the catalog and what the servers read at runtime live in the platform DB. bodhi-go-server already has this connection — database/platform.go's PlatformDB — today used only for a shadow path; this design promotes it to the model source.

There are exactly two platform databases, dev and prod (verified on the server):

The design keeps that isolation and bridges it only through the pipeline:

• The catalog = the model tables in the platform DB where the AI team authors a model, written once in dev. This is the source of truth.
• Each environment reads its own platform DB. Dev servers read bodhi-api-platform, prod servers read bodhi-api-platform-prod — isolated, no cross-environment dependency.
• The pipeline (Jenkins) is the only bridge between dev and prod.

                    ┌──────────────────────────┐
   AI team ────────►│   CATALOG (author once)  │   ← source of truth
   dashboard        │  family, version, config │
                    └──────────────────────────┘
                          │                │
             "Promote to test"      "Promote to live"
             (dev Jenkins job)      (prod Jenkins job)
                          │                │
                          ▼                     ▼
                 ┌───────────────────┐  ┌─────────────────────────┐
                 │ bodhi-api-platform│  │ bodhi-api-platform-prod │  ← each server reads
                 │       (dev)       │  │         (prod)          │     ONLY its own DB
                 └───────────────────┘  └─────────────────────────┘

Why this matters: config is authored once and pushed byte-for-byte to dev, then to prod. Dev and prod can't drift, because prod isn't edited separately — it receives the same catalog entry dev already validated.

4The pieces

1. The Catalog — a set of tables in the platform DB. Every model is one entry: family, version, config, test/live status, public/custom, where its files are, and which environments it's deployed to.
2. The Dashboard — a web screen for the AI team. Upload, configure with fields/dropdowns (no editing JSON files), promote test→live, assign to customers, hit Deploy. Also shows "which model runs where."
3. Storage & packaging (KitOps / OCI) — each model is packaged as a ModelKit (an OCI artifact bundling weights + baseline config + tokenizer) with the KitOps kit CLI and pushed to an OCI registry we already run (ECR / Azure ACR). Content-addressed, SHA-256 tamper-evident, versioned by tag/digest, Cosign-signed, with an SBOM/audit trail. Replaces "loose encrypted files on a mount." Weights are still encrypted — see §8.
4. The Pipeline (Jenkins) — already exists. On Deploy it syncs the catalog to the right environment's DB and rolling-restarts the servers.
5. The Server (bodhi-go-server) — at startup reads its environment's platform DB (via PlatformDB), decrypts only the models it's allowed to run, and loads them.

5The catalog schema

All greenfield — none of this exists today.

model_families        v1/v2 grouping. "which model belongs to which"
  id, name (e.g. "hi-banking"), language, description,
  visibility ─ public | custom       ← set on the FAMILY: public = all customers,
                                        custom = only customers in customer_families.
                                        Every version inherits it.

models                one version of a family = one catalog entry
  id, family_id, version, model_name (unique),
  status  ─ draft | test | live | deprecated | retired   ← lifecycle (see 7b)
  storage_uri   (ModelKit ref: registry repo@sha256-digest — immutable; see 4, 7b)
  encrypted (bool), key_id, checksum
  config jsonb  (tunable OVERRIDES: endpointing/silence/hotwords — baseline ships in
                 the ModelKit; this layer is what the dashboard edits. See note below.)
  created_by, timestamps, deleted_at              ← soft-delete only; row is never dropped

model_deployments     THE MISSING MAP: what runs where, since when
  model_id, environment (dev|prod), state (deployed|removed),
  deployed_at, deployed_by, removed_at, removed_by

customer_families     custom-family entitlement (only for visibility=custom)
  customer_id, family_id, granted_by, granted_at

Two tables kill the two biggest pains directly:

• model_deployments = the transparency/onboarding answer. "Which model runs where?" becomes one query. A new hire reads a table, not tribal knowledge.
• customer_families = per-customer custom families, extending our coarse products_enabled (asr/tts) gate down to the family level. Grant a customer a family once and every future version is automatically theirs.

6The concepts, in plain words

• Family = a group of models that do the same job over time (e.g. "hi-banking"). Its members are v1, v2, v3…
• Version = which one inside the family. Keep the old one running while testing the new one.
• Test vs Live = NOT a single switch (dev and prod are different databases). It means which environment a model has been promoted to: Test = promoted to dev (bodhi-api-platform); Live = promoted to prod (bodhi-api-platform-prod). Lines up with the split Jenkins jobs (Jenkinsfile-stt-dev vs -stt-prod).
• Dev vs Prod models = not two lists — the same model promoted from dev to prod. One catalog, no divergence.
• Public vs Custom = a property of the family, inherited by all versions. A public family is available to every customer; a custom family is tied to specific customers (in customer_families). This answers who's allowed the lineage; deployment answers which version runs.

7From "trained" to "live" (the main story)

1. Upload / package. AI team drops the model on the dashboard. Behind it, the weights are encrypted and packaged as a ModelKit, then pushed to the OCI registry (kit pack + kit push). (No more "copy to mount + push config to git + ask devops.")
2. Configure. Set the model's override config (endpointing / silence / hotword knobs — layered on top of the ModelKit's baseline), pick its family and version. It starts in Test.
3. Promote to Test. Dev pipeline writes it to bodhi-api-platform and rolling-restarts dev. Prod untouched.
4. Promote to Live. The prod pipeline writes the same entry to bodhi-api-platform-prod and rolling-restarts prod.
5. Assign (optional). If the model's family is custom, tick the customer(s) (customer_families). Otherwise the family is public.
6. Done. Dashboard shows: hi-banking-v3 — in Test since Mon, in Live since Wed.

The whole thing is self-serve for the AI team. Devops isn't in the loop for a routine model change anymore.

7bTaking a model down (deprecate → undeploy → retire)

Retiring a model is three distinct actions, not a delete button. There is no hard-delete in the normal path — nothing is ever destroyed, only archived.

1. Deprecate (graceful end-of-life, still serving). Set status = deprecated. The model stays loaded and servable but is hidden from the assignable list. Reversible. The normal "v3 is out, wind v2 down" step.
2. Undeploy (stop running it in an environment). Flip its model_deployments row to state = removed → it drops out of that env's load list → pipeline rolling-restarts → pods come back without it. Dev and prod undeploy independently; the row stays.
   Guard: before undeploy, check customer_families + recent usage. If customers still depend on it, block or force-with-confirmation; for a custom family, un-assign its customers first.
3. Retire (stop shipping it, keep the record). Once undeployed from all environments: set status = retired + deleted_at. In SaaS the catalog DB row is the load list (loadModelsFromDB), so this alone drops the model on the next restart — no mount edits, no index.json (that's the benchmark/on-prem loader). The ModelKit stays in the registry at its immutable digest — nothing is deleted, so storage_uri still resolves and it's fully recoverable (optionally re-tag it into an archived/ repo). The row is kept forever so logs still resolve. Restore = clear deleted_at, redeploy — the pod re-pulls the same digest.

Sequence matters: undeploy (stop serving) → flip the catalog row. The ModelKit is immutable in the registry, so there's nothing to move or delete on a live pod — a retired model simply stops being pulled on the next restart.

Lifecycle end-to-end: draft → test → live → deprecated → undeploy (per env) → retire.

8Encryption, in plain words

Every model file is locked with a key before storage. The server unlocks it only in memory at startup — the unlocked version never touches disk.

• The key hides the bytes; it does NOT decide who's allowed. Access is decided by the catalog + dwarpal. A customer using a custom family is blocked at the authorization layer, not just by encryption.
• We already built the lock/unlock code for on-prem. We reuse it exactly. Only the key source differs:

                        on-prem            SaaS (this system)
  DecryptKey    ◄──   signed license   │   Azure Key Vault (pod managed identity)
  AllowedModels ◄──   license payload  │   catalog: customer_families table (via dwarpal)
  model list    ◄──   index.json embed │   catalog DB
                     ─────────────────────────────────────────
                        SAME decrypt + load + recognizer code below

InitializeAllModels already takes the decrypt key and allowlist as arguments — it doesn't care where they came from. So supporting SaaS is a config seam, not a fork.

9What the server does at runtime

At startup: ask its platform DB (PlatformDB) which models to run (status = live; each row's storage_uri is a ModelKit digest) → pull + unpack those ModelKits from the registry (kit unpack, or reuse the cached layer if the digest is unchanged) → get the unlock key (Key Vault in cloud; license on-prem) → decrypt in memory and load → start serving.

Per request: dwarpal checks the customer is allowed the product and, if they asked for a model in a custom family, that the family is granted to them (customer_families) → transcribe.

10Reloading across many pods → rolling restart

Each pod holds models in its own memory, so we let Kubernetes reload all pods via a rolling restart (kubectl rollout restart, already in the pipeline). The dashboard "Deploy/Reload" button just triggers that job — no image rebuild for a model-only change.

 Start: [pod1 old] [pod2 old] [pod3 old]   ← all serving
   │  k8s starts a new pod, waits till it's READY (models loaded), then kills one old
   ▼
        [pod1 NEW] [pod2 old] [pod3 old]
        [pod1 NEW] [pod2 NEW] [pod3 old]
        [pod1 NEW] [pod2 NEW] [pod3 NEW]   ← done, zero downtime

Old pods keep serving throughout → no dropped requests. True in-process hot reload is a possible future optimization only if model-load time ever makes restarts too slow — not needed now.

11What we reuse

So this is not a from-scratch mega-project.

• Encryption + in-memory (memfd) decrypt — already written on the on-prem branch (cmd/encrypt-models, utils/model_crypto.go). Reused as-is.
• FilterAllowedModels — the per-customer gate, fed from the catalog instead of a license.
• loadModelsFromDB already exists — InitializeAllModels already loads from a DB for every non-benchmark environment; the SaaS change is mostly pointing it at PlatformDB instead of the old database.DB.
• KitOps + our OCI registry — model packaging, versioning, signing, and storage come from KitOps (kit) into a registry we already run (ECR / Azure ACR). We don't build a bespoke model store.
• The Jenkins pipeline — exists; finish the "update config + rolling restart" path.
• dwarpal auth — already checks customers; add one small custom-family check.
• The model config format — the knobs we already store; the tunable ones move from git files into the catalog DB (as overrides), the baseline into the ModelKit.

Genuinely new: the catalog tables, the dashboard, and wiring the dots (including kit pack/push on upload and kit unpack at boot).

12Build order

Each step is useful on its own.

You can stop after any step and still be better off than today. Step 1 alone kills the "nobody knows what's running" problem.

13Open decisions

1. Retire bodhi_asr_configs git repo once the catalog exists, or keep it as an export/backup?

14Related: the LC + finetune data platform

This catalog is one half of a lifecycle; the other half is the event / LC / finetune platform. They connect through one thing: every model here has an immutable model_id + version, and every ASR/TTS request logs that id into the ClickHouse event lake. So the platform can attribute quality to an exact model version, and the AI team finetunes the right one.

   model catalog (this doc)  ─ deploys a version
        └─► ClickHouse event lake  ─ what the model produced (model_id + version)
              └─► LC reviews, corrects (bodhi-lc)
                    └─► AI team finetunes ─► new version uploaded to the catalog  ↺