Learning Product Design | B2B | Learning Data Analysis
Session Scope
Session Scope turns recordings of online tutoring sessions into a transcript, labels the action in every sentence, and judges whether those actions help students learn. I was the researcher on the team at Varsity Tutors, where it stayed a prototype.
[The Problem]
Varsity Tutors approached us with the problem that although they have rich and dense audio recordings of tutoring sessions conducted on their platform, they are not gathering enough insights from those recordings.
Current Data Focus
Data that truly helps students
- No show rate
- Time on task
- Client Churn Rate
- Feedback Delivery
- Scaffolding Quality
- Answer Alignment
[How it works]
- AudioTutoring session recording
- TranscriptOpen-source transcription, hosted locally
- Labeled sentencesOne of 30+ action tags per sentence
- EvaluationProductive or counterproductive, per learning science
- DashboardFindings sorted by the team that needs them
[The dashboard]
[Challenge 1]
Since most students are teenagers, our analysis is restricted to session transcripts only. No video recordings are available.
[Strategy 1 - Transcription]
Research an open-source library that transcribes audio to text and can be hosted locally, then automate transcription so further analysis can happen.
[Challenge 2]
How to summarize and monitor the actions happening in tutoring with only reading the transcribed text of the session?
[Strategy 2 - Discourse Analysis]
With research, we discovered a labeling technique named “Discourse Analysis.”
Discourse analysis is a strategy where you take one sentence out of the transcript, then label it with an action that is most fitting for the action happening in that sentence. For example, “I don’t get this, how did you know what formula to use here” is a sentence that you can label “QUESTION”.
With this strategy, we can label all sentences in our transcript with one out of over thirty tags to elicit the action happening in each and every moment of the tutoring session.
“I don’t get this, how did you know what formula to use here”QUESTION
[Challenge 3]
Labeling each and every sentence of a transcript is a lot of work. How do we automate this process?
[Strategy 3 - Automation]
We trained our own BERT model to do the labeling. We hand-labeled over 12 hours of transcript so that every tag had enough example sentences, then fed those sentences and their discourse analysis tags into the model.
The model learned to choose the appropriate tag for each sentence, so new transcripts get labeled automatically instead of by hand.
[Challenge 4]
Now that we know what happened in the sessions, how do we know whether they were helpful?
[Strategy 4 - Evaluation with audience in mind]
There are so many measurements of whether a session went right or wrong. To make our insights immediately valuable, we based our evaluation features on existing teams at Varsity Tutors.
The first step is to figure out our audiences: who are the users for this product?
The next step is to ask each user: what matters the most for you?
We developed our specific features trying to answer these questions. Features like sentiment analysis, scaffolding and pedagogy quality, and rule violations are included in our final product to address their primary concerns.
[Before and after Session Scope]
Before, reviewing a session meant scrubbing through the audio and typing notes by hand. After, each team sees the findings it cares about. The screens below are a recreation of the review tool, not a capture of the live product.
Student struggled with quadratic factoring early in the session; recovered confidently after the example walkthrough at 35:00.
Compliance management
- Generate audit-ready session summaries
- Reduce manual review time
Wellbeing Support
- Detect frustration spikes automatically
- Highlight positive interactions
Pedagogy quality
- Highlight scaffolding and teaching trends
- Generate session trends
[My role]
I was the researcher on the team, and the work had two sides.
Product research. I talked to the teams at Varsity Tutors to find out what each of them needed from the dashboard, and those conversations decided which features we built.
Technology research. I learned about discourse analysis and applied it to our product.