LLM engineering · Grounding · VakSutra AI
How we stop an AI coach from inventing feedback
In VakSutra AI, code computes the score and the LLM only explains it — with verbatim quotes, real timestamps and strict rules. Here is how the split works and what we would add next.
By the CognifySystems engineering team · · 6 min read
The problem: confident feedback that isn't true
VakSutra AI is our AI speaking coach. You speak for up to two minutes, and it tells you how it went. The easy version of this product sends a transcript to an LLM and asks for feedback. It produces fluent, confident text — and some of it is made up: a pause that never happened, “you said ‘um’ often” when you said it once, a quote you never spoke.
For a coaching product that is fatal. The first time a user sees feedback about something they didn't do, they stop trusting all of it. So we designed the system around one rule: the LLM is never the source of facts.
Split the work: code measures, the LLM explains
Everything that can be measured is measured in code before the LLM is involved. Whisper gives a transcript with word-level timestamps. Pitch is extracted with librosa, and a small Rust extension computes loudness and pauses. Face analysis in the browser produces gaze and expression signals. All of it is stitched onto one millisecond timeline of events.
The score itself is a fixed formula — 35% Fluency, 20% Language, 20% Vocal, 25% Presence — computed from those signals. The LLM receives the score as an input and is told it may not change it. Its job is the part code is bad at: explaining what mattered most, in plain language, and showing how to say it better.
Give the model evidence, not raw data
The model never sees audio or video. It gets a structured context: the transcript, the computed metrics, the list of detected events with their timestamps, and derived labels (for example a pacing label such as “Rushed”) that it is told to reuse exactly rather than re-derive.
User-controlled text — the speaking topic — is wrapped in explicit delimiters and marked as untrusted, so instructions hidden in a topic are treated as plain text, not commands.
Rules the coaching must follow
The prompt carries a short list of hard rules. Each one exists because we saw the failure it prevents:
- Quotes in “before” corrections must be exact words from the transcript.
- Timestamps may only reference events in the provided event list.
- Frequency words (“often”, “kept”) are only allowed when the actual count supports them. Once is “once”.
- Describe observable behaviour only — “broke eye contact”, never “seemed nervous”. No diagnosing anxiety or mental state.
- Judge how the idea was delivered, never the idea or opinion itself.
- Name exactly one main bottleneck, so the user knows what to fix first.
- Rewrites must be real spoken words — no “[pause]” or “[breathe]” shortcuts.
Structured output, and safe failure
The model returns JSON in a fixed shape — review, main bottleneck with evidence, corrections, and a “proof moment” (the user's original sentence next to a stronger version). We parse it defensively, tolerating stray code fences or a preamble, and validate it against the schema; anything that doesn't fit is rejected rather than shown.
Not every input needs an LLM. If a transcript has fewer than ten words there is nothing meaningful to coach, so the app returns a short report instantly without a model call. And because the session is saved before coaching starts, a slow or failing model never loses the user's recording or score.
What we would add next
Today the quote and timestamp rules are enforced through the prompt and the output schema. The natural next step is to verify them in code after generation: check that every quoted “before” string appears in the transcript, and that every cited timestamp matches a detected event, and drop or regenerate anything that doesn't. Prompt rules make violations rare; a programmatic check makes them impossible to show.
Takeaways for any AI product
The pattern generalises beyond speech coaching. Anywhere an AI feature makes claims about a user's data — a report, a summary, an assessment — the same structure helps:
- Compute facts and scores deterministically; give them to the model as inputs.
- Pass evidence with identifiers (quotes, timestamps, IDs) the output must reuse.
- Write rules for the specific failures you have observed, not generic “be accurate” instructions.
- Use structured output and reject what doesn't validate.
- Verify claims against the source in code where you can.