An AI speaking assistant that reads a live Zoom conversation
Not a separate window you re-explain yourself to: the server already holds the context of the conversation in progress.
- Zoom RTMS
- WebSocket
- Webhook
- SSE
- LLM API
- JSON Schema
- AWS S3
Specification
- Service
- Zoom-based English conversation sessions
- Period
- 2025 – 2026
- Inbound
- Zoom RTMS over webhook and WebSocket
- Outbound
- SSE, server to the user's screen
- Role
- Realtime pipeline, session state design, LLM integration and response validation
Context
The user is mid-conversation with a real person. Opening another window and re-explaining the situation from scratch means the conversation has already moved on.
For the help to be useful, the AI has to already know what has been said.
Approach
I integrated Zoom RTMS, receiving participants' live speech into the server over webhook and WebSocket.
Utterances are collected per participant so it is clear who said what, then aggregated into per-session state. By the time a request arrives, the server already holds the conversation up to that moment.
Server state and the data the user needs are pushed to the browser over SSE. Inbound (WebSocket) and outbound (SSE) are different problems in different directions and are kept separate.
On a request, the current transcript and context are composed into a prompt context and sent to the LLM API.
Free text from the LLM is not trusted as-is. A JSON Schema validates the response, so a missing required field or an unexpected shape is caught in the service layer. An LLM is not a special kind of thing; it is an external system that needs verification.
The user's recent correction history goes into the context as well, so a mistake repeated from earlier sessions can be recognised as a pattern.
Information that must be generated is separated from facts the system has to guarantee. The LLM handles language understanding, grammar and phrasing analysis, and correction judgement; actual dates, verbatim past utterances, user records and confirmed corrections are re-joined from the database. That separation reduces the likelihood of hallucination.
System
WebSocket in, SSE out, with schema validation between them.
Speech from the Zoom session arrives over RTMS through webhook and WebSocket and is aggregated into per-participant session state. On a request, transcript and correction history are composed into a prompt, the LLM is called, the response is validated against a JSON Schema, and the result is pushed to the user's screen over SSE.
Boundaries
- No shareable measurement exists for this work, so no performance or accuracy figure is claimed.
- No model was trained or fine-tuned; what was used is an LLM API.
- I do not claim hallucination was eliminated. Re-joining facts from the database reduces its likelihood.