Event Logging & the Data Platform

How every ASR and TTS request is recorded in one analytical store (ClickHouse), and how that same store grows into the LC-review and finetuning platform. The idea and the overview logic — not an implementation spec.

1Why this exists

Today the picture is broken:

• ASR writes transcripts to the old streaming-asr Postgres (streaming_logs), via a fire-and-forget path that silently drops rows. Not go-live safe, and tied to a DB we're retiring.
• TTS (bodhi-tts-server) records nothing at all — it only reports usage to dwarpal and streams audio.

So we have no reliable, queryable record of what the models actually produced — which is exactly what LC and the AI team need to find issues and finetune.

The fix: every ASR and TTS request emits one immutable event into ClickHouse — a columnar store built for huge append-only volume, cheap compression, and fast slice-and-dice. Billing stays with dwarpal; LC labels come later in their own store. ClickHouse is only the event lake.

2The one mental model

   ASR / TTS request                                          later
 ┌──────────────────┐    emit event   ┌───────────────┐   ┌──────────────────────┐
 │ bodhi-go-server  ├────────────────►│  ClickHouse   │──►│ LC dashboard         │
 │ bodhi-tts-server │  (via a queue)  │  event lake   │   │ (find issues)        │
 └──────────────────┘                 │ asr_events    │   ├──────────────────────┤
        │                             │ tts_events    │   │ Finetune dataset     │
        │ report usage                └───────────────┘   │ builder (AI team)    │
        ▼                                    ▲             └──────────────────────┘
   dwarpal (billing) ── platform DB          │  join on transaction_id
                                    ┌─────────┴─────────┐
                                    │ bodhi-lc Postgres │  ← LC scores + human
                                    │ (mutable labels)  │     corrections (later)
                                    └───────────────────┘

• ClickHouse = the immutable log of what was produced (append-only).
• dwarpal / platform DB = the money (already built — don't duplicate).
• bodhi-lc Postgres = the mutable human labels (built later; joins by transaction_id).

3Core principles (what makes it reliable)

1. Emit through a durable queue, not a bare goroutine. Producer → Azure Event Hubs (Kafka API) → ClickHouse Kafka-engine table → materialized view → MergeTree. No custom consumer, batched inserts, replayable. (Lighter alt: ClickHouse async_insert with an app-side disk buffer.)
2. Logging is independent of everything else. The event is emitted even if audio save fails (audio_ref = null) and regardless of billing.
3. Never swallow errors. A failed emit increments a metric and alerts.
4. Idempotent on transaction_id. Retries/replays don't double-count.
5. Immutable only. Mutable things (LC scores, money) live elsewhere.

4What we store

asr_events — one row per transcription request

tts_events — one row per synthesis request

Engine sketch (both): MergeTree, PARTITION BY toYYYYMM(request_ts), ORDER BY (customer_id, model_id, request_ts), TTL for retention. Big payloads (audio, and optionally very long transcripts/text) live in Blob, referenced by *_ref.

5Later: the LC layer

The LC team reviews calls to find where ASR/TTS went wrong. That produces mutable, human-owned data — so it gets its own Postgres DB (bodhi-lc), keyed by transaction_id, joined to the ClickHouse event at read time:

• lc_reviews: transaction_id, reviewer, score, comments, error_tags, reviewed_at
• corrections: transaction_id, human-corrected reference transcript (ASR) / correct pronunciation (TTS) — this is the finetuning fuel
• dataset_members: which reviewed items belong to which finetuning dataset

The dashboard = "browse events from ClickHouse, filter to suspicious ones (low confidence, long lag, a given model_version/language/customer), open one, score/correct it → write to bodhi-lc."

6Later: the finetune platform

Building a training set becomes two queries joined on transaction_id:

1. ClickHouse — find the cohort: "all hi-banking-v3 calls last month with confidence < X, for customer Y".
2. bodhi-lc — pull the human corrections for those transactions.
3. Export (audio_ref, model output, corrected reference, tags) → the AI team finetunes → ships a new model version via the model catalog.

That closes the loop with the model catalog:

  model catalog deploys a version
        → ASR/TTS emit events (ClickHouse)
        → LC finds its issues (bodhi-lc)
        → AI team finetunes
        → new version uploaded to the catalog → deploy   ↺

The catalog gives every event a precise model_id/model_version, so the platform can say "v3 regressed on Hindi numbers vs v2" — and target the finetune.

7Build order (rough)

Steps 1–3 are the go-live requirement. 4–5 are the platform that grows on top of the same events.

8Open decisions

1. ClickHouse: managed (ClickHouse Cloud on Azure) vs self-hosted. Managed is faster to go-live.
2. Queue: Event Hubs + Kafka-engine table vs app-side async_insert.
3. Transcript / input_text: inline column vs Blob pointer. Can start inline, move later.
4. Retention / TTL per table before volume grows.