An operator's lab that replays real sessions to re-measure prompt quality
Running a fresh Zoom session for every prompt change is wasteful, and invented test sentences are not real conversation. Stored sessions are replayed up to a chosen moment and compared under identical conditions.
- AWS S3
- LLM API
- CI/CD
Specification
- Service
- Spker
- Period
- 2025 – 2026
- Data
- Recordings and transcripts stored in AWS S3
- Role
- Replay tooling design and development, quality verification procedure
Context
Change the prompt and the output changes. But running a fresh Zoom session to check each change is not something you can repeat.
Invented test sentences are no better: they lack the context and the error patterns that appear in real conversation. Evaluation data unlike production produces results unlike production.
Approach
I built a lab that loads a past session from the recordings and transcripts in S3 and replays the conversation up to a chosen point — for example, reconstructing the session state of a 30-minute session as it stood at 17 minutes.
From that state the AI is asked for help, showing what it would have produced in that real context. After a prompt change, the same session at the same moment is run again and the two responses compared.
Speaker matching — tying each utterance in the transcript to an actual participant — is confirmed once by an operator, stored, and reused when the report is generated, which removes the repeated manual pass over the same information.
The post-session report also draws on the transcript and the AI analysis, and its correction history feeds back as personalisation context for the next session.
Because the AI behaviour and the prompts change often, I set up CI/CD so improvements could be deployed to the running service repeatedly rather than occasionally.
System
Same session, same moment. The only change is the prompt.
A real session's recording and transcript are stored in S3. The lab replays it to a chosen moment to build the session state, runs the prompt against it, compares responses before and after, and after a revision returns to the same session at the same moment to run again.
Boundaries
- No shareable measurement exists here, so no figure is claimed for the size of the quality improvement.
- The lab is an internal operator tool and is not exposed to users.