Q&AI: retrieval-augmented search that lands on the exact second.

Architecture

Two paths share one store. Offline, videos are discovered, transcribed with word-level timing, de-duplicated, checked for playability and embedded into one Chroma collection per person. Online, a question goes through query expansion, fused vector retrieval, LLM answer selection and timestamp alignment before the browser starts the YouTube embed at the matched second.

1 · Ingestion

Everything downstream depends on knowing when each word was spoken, so transcripts are stored as segments of words with start and end times:

[{"start": 16.22, "end": 20.88, "text": "You never know what's going to cause the next…",
  "words": [{"word": "You",   "start": 16.22, "end": 16.66},
            {"word": "never", "start": 16.66, "end": 16.84}, …]}]

• YouTube captions. The json3 format carries per-word offsets, so timings come without downloading audio; they agree with Whisper to a median of 0.05 s.
• Hand-made captions, merged in. Aligned to the recognition captions with difflib so words gain punctuation and capitals but keep their timings.
• Whisper, when there are no captions. Audio is cut into 10-minute pieces and transcribed in parallel with whisper-large-v3 on Groq (about 6 s per 10 minutes), then stitched back with monotonic word times.

2 · Indexing

Words are packed into chunks of up to 1000 characters, with the word timings stored in each chunk's metadata. Each person is a separate Chroma collection with its own embedding model:

Indexing is incremental: on startup each collection embeds only new videos, one batched call per video, and drops chunks whose transcripts were removed.

3 · Retrieval

Query expansion. Questions and answers are phrased differently ("Why did you start SpaceX?" vs "I thought the odds of success were less than 10%"). A small LLM (gpt-oss-20b) writes five answers in the person's voice and transcript language, and these act as extra queries.

Weighted reciprocal rank fusion. The question and its rewrites each return their top 8 chunks, fused with the original question weighted three times as heavily:

Sweeping \(w_{\text{original}} \in \{0, 1, 2, 3, 5\}\) on 150 questions, \(w = 3\) lifted hit@1 from 58% to 64% (hit@5 89%), replacing a vote count whose top-5 was full of ties.

Boundary-aware context. Each of the top 5 chunks is passed with the chunk that follows it, fixing answers that start right at a chunk boundary ("Why did you start SpaceX?" had returned just "Yes. And to").

4 · Answer selection

The answer model gets the question and five numbered excerpts and returns <excerpt>: <verbatim passage>, or a sentinel when nothing fits. It must pick the excerpt that answers the question, start at the first sentence that does, and copy contiguously without edits. Prompts and Groq-hosted models were compared on the LLM-judged harness below: qwen3.8-27b matched the best model's quality at about a seventh of its latency, with gpt-oss-120b as a fallback on rate limits, timeouts or errors.

5 · Timestamp alignment

The model returns text, not times, and doesn't copy perfectly. The first aligner matched the quote's first sentence at its earliest occurrence; answers often open with "Yeah." or "Okay.", so 11.5% of clips started more than 5 s early. The current one matches 12-word anchors at both ends of the quote, maximised over every position, with near-ties broken by a 48-word window:

# simplified
def align(quote, words):
    w = [normalize(x.word) for x in words]           # lowercase, strip punctuation and accents
    first, last = split_on_ellipsis(quote)           # "…" means the model skipped a part
    i, score = best_window(w, first[:12], tiebreak=first[:48])
    j, _     = best_window(w, last[-12:], tiebreak=last[-48:], align="end")
    if score < 60:
        return None                                  # caller tries the other retrieved chunks
    return time_of(words, i, "start", step=+1) - 0.3 # nearest word that has a timestamp

Benchmarked on 600 real transcript passages, verbatim and with perturbations that mimic the model:

6 · Evaluation

Every change was decided by an evaluation harness, not spot checks. It measures retrieval (hit@1, hit@5, MRR on 150 LLM-generated questions with a known source chunk), end to end (does the played clip fall inside the source chunk) and answer quality (two judges from different model families score directness 1–5, on generated plus real suggested questions).

The key finding: after the retrieval fixes, end-to-end accuracy didn't move. The bottleneck had become whether the right chunk reaches the top 5 and whether the model picks it, which moved the effort to answer selection and context.

7 · Performance

8 · Serving and operations

Flask on waitress, with Chroma collections shared across request threads. Searches are rate-limited per visitor and site-wide, since each one costs several LLM calls. People are configuration, not code (transcripts, embedding model, language, persona, suggested questions), so adding the Greek collection needed no new code paths. Lint, lock-file checks and unit tests run on every push in GitHub Actions.

Next steps

• Cross-encoder re-ranking of a wider candidate set, since most remaining errors are the right chunk missing from the top 5.
• Sentence-index answers: the model returns a sentence range instead of copying text, giving exact start and end times with no fuzzy matching.
• Speaker attribution, so a clip never starts on the interviewer's question.