Methodology
How we grade the professor. Only what Scott predicts on the record, held to the deadline he set.
The source
What counts as a prediction
Counted
- Forecasts Scott makes during the predictions segment
- Every numbered episode of Pivot, plus year-end predictions episodes
- Claims about something that hadn’t happened yet when the episode aired
Not counted
- Offhand forecasts from elsewhere in the episode
- Kara’s predictions, or guests’
- Interviews, bonus episodes, and cross-promos
- Jokes, and predictions Scott attributes to someone else
Sticking to the predictions segment keeps the bar fair: these are the calls Scott makes knowing they’re predictions, not every passing remark in an hour of conversation.
The process
From podcast to grade
- 1
Listen to the predictions segment
Regular Pivot episodes end with Kara asking Scott for a prediction. We transcribe the closing minutes of every numbered episode, plus the dedicated year-end predictions episodes in full.
- 2
Pull out Scott’s forecasts
An AI model reads the segment and lifts each forecast Scott makes, word for word. Every quote links back to the exact moment in the transcript and audio.
- 3
Pin down the claim
Each quote is restated as one claim that is either true or false, with a deadline. If Scott didn’t give a time frame, we infer one (usually a year) and mark it as inferred.
- 4
Grade it when it’s settled
Once a prediction is due, an AI model researches what actually happened on the web and records a verdict with its sources, plus the date the outcome became known. Open predictions are also checked monthly, so one that comes true early is graded when it happens. Editors can review and overrule any grade.
The verdicts
What each grade means
- Came true
The evidence shows it happened, as stated, by the deadline.
- Didn’t happen
The deadline passed and it didn’t happen.
- Too early to call
Due, but not settled yet, like a deal announced but not closed. We check again later.
- Not due yet
The deadline hasn’t arrived.
- Didn’t apply
An “if X, then Y” prediction where X never happened.
- Can’t be graded
Too vague to settle, or the facts can’t be found in public sources.
Only came true and didn’t happen count toward Scott’s record. Everything else is shown, but left out of the score.
The fine print
Where this can go wrong
- Transcripts are automatic. Words and speaker labels come from software and can be wrong. The audio is always one click away.
- AI makes mistakes. Extraction and grading are done by a model, and each grade says so. Editors review grades, and a human grade is never overwritten by the model.
- Restating takes judgment. Turning a spoken remark into a checkable claim means choosing words. We stay close to what Scott said and show the original quote next to every claim.