Our Updated Media Processing Pipeline
How we cut timestamp-matching costs 15x and made it 40% faster.
A note on timing. We have a backlog of engineering blogposts we re working through. Most of the work described here took place in April and May 2026, and we are only writing it up now, in October 2026.
In a previous post we described how we located the exact start and end timestamps for every Congressional Record statement in floor video. That pipeline ran in three stages: fast exact matching, LLM-assisted fuzzy matching on short snippets, and—when both of those failed—a final fallback that passed the entire transcript to a state-of-the-art LLM (the early GPT-5 series in our first versions). The rough breakdown of how many target sentences each stage resolved is below.
Even as we shipped this pipeline, we were already thinking about how to improve Stages 2 and 3. Stage 2 is highly accurate and uses few tokens, but the method itself is complex and has several sub-phases. Stage 3, while used on only a small fraction of statements, is inefficient: it passes the entire transcript to a state-of-the-art (SOTA) LLM, because smaller LLMs were not accurate enough for production. That makes it costly, both because SOTA LLMs are more expensive per token and because the full transcript can easily run into the hundreds of thousands of tokens. Senator Cory Booker's 25-hour filibuster, for example, produced 13,000 sentences and more than 640,000 tokens. At that length a transcript may not fit in the model's context window at all.
When we started looking for a better approach to these two stages, we took a cue from how we work. We debug the media pipeline often—most heavily during the core design work, but also as corner cases come up in daily processing—and it usually means reviewing failure cases by hand. The way we do that is simple: we open the transcript and use ordinary text search to look for a snippet of the target statement. Because this is a fuzzy matching task (see below), the first search doesn't always produce a hit, so we search again with a different phrase. Doing this by hand, we quickly developed a small set of simple, intuitive search rules.
Our first idea was therefore to replace Stages 2 and 3 with an agentic search procedure that mimicked our own manual approach, adding a few tools we thought the model would need for better transcript navigation and to record the timestamps it found. This involved a set of six tools that let the model work the problem in a loop: it could search the transcript for a pattern, read a specific range of lines, pull more context from the target statement, set a start or end timestamp, read the start timestamp it had already set, and signal when it was done. But after much experimentation this turned out to be too complex for the model to learn well, and ultimately unnecessary.
We then tried a simpler approach that split the work across two models—a model focused on the search task and a model focused on the timestamp identification task—and used a few tricks, including improved labeling and a fine-tuning dataset we built with a custom action-recording tool.
We eventually moved away from that approach too, in favor of a simple deterministic algorithm paired with a medium-sized model (currently DeepSeek V4.1 Flash) for final timestamp identification. The algorithm searches the transcript for every n-gram of 5 to 12 words from the target statement and keeps the matches from the rarest n-grams (the fewest matches, with longer n-grams winning ties). We then pass the target and the transcript lines surrounding each match to the model, which returns the timestamp in the transcript that matches the target statement. We do this for both the opening and closing sentences of the statement, which gives us the start and end timestamps. We then cut the audio and video using those timestamps. This approach vastly simplifies the timestamp matching logic, reduces costs by about 15x overall and 40x on the hardest statements, and is about 40% faster.
A Reminder of Why Timestamp Matching Is Hard
Before describing the new fallback, it's worth restating what makes timestamp matching surprisingly difficult, because it drives the design choices that follow. At the heart of the task is matching two versions of the same speech: the official written Congressional Record released by Congress (an edited version of every statement made on the floor) and a transcript of the floor audio. The Congressional Record is edited for the printed page, official reporters clean up grammar, drop filler, and tidy sentences, and members may revise and extend their own remarks before publication. The transcript is produced by re-transcribing the floor audio, so it includes contractions, misheard proper nouns, restarts, audio drops, and asides that differ from the written record. These differences can be small or large, and there is no pattern to them that can be applied programmatically.
A concrete example makes the gap obvious. Here is the same passage from a statement by Rep. David Schweikert as it appears in each source. The differences are numerous; a few are highlighted:
First Attempt: An Agent with Search Tools
Our first replacement for Stages 2 and 3 was an agent. Its task was to find the target sentences in the transcript and return their timestamps. We gave a general-purpose model (GPT-5.2 with medium reasoning) a system prompt describing the task and a set of tools that mirrored our manual workflow:
- Search the transcript for the target sentences with a regular expression, returning each matching line with its timestamps and up to 12 surrounding lines.
- Read transcript lines by line number.
- Expand the target with additional sentences from the Congressional Record statement, to give the agent more text to search for.
- Set the start or end timestamp, given in milliseconds. Only a timestamp that matches the start or end time of a transcript line is accepted.
- Read the start timestamp if the statement's start timestamp had already been found, so the end timestamp could be placed after it. This helps when the model is unsure where the statement ends, or when it finds the closing sentences more than once. That happens because floor proceedings are sometimes re-aired later the same day, so the same statement can appear twice in one transcript.
- Mark the task as finished.
The model called these tools in a loop until it marked the task as finished. We did no task-specific training for this version. It failed in four consistent ways that we could not fix with prompting:
The last of these is the main reason we turned to fine-tuning. Models—even the largest state-of-the-art models at the time of testing, such as GPT-5.2—often called the search tool repeatedly, even after a search result already contained the line that matched the target. At that point the correct action is to commit that line's start or end timestamp.
The table below shows GPT-5.2 doing this while looking for the start of a statement by Rep. Ryan Zinke on January 10, 2023. Its first search returned line 1487, "Mr. Speaker, I rise today in support of a select committee to investigate the weaponization of the federal government," which is the first sentence of his statement almost word for word. Instead of setting the start timestamp, it spent its remaining seven steps reading other lines and searching again, and it ran out of steps without ever setting one.
| Step | Action | Result |
|---|---|---|
| 1 | search_pattern "weaponization of the federal government" | returned line 1487, the start |
| 2 | get_lines 1208–1245 | read other lines |
| 3 | get_lines 1214–1236 | read other lines |
| 4 | search_pattern "proudly served as the 52nd secretary of the interior" | no results |
| 5 | search_pattern "52nd secretary of the interior" | no results |
| 6 | get_lines 1216–1234 | read other lines |
| 7 | search_pattern "deep state" / "duly elected member" | returned line 1487 again |
| 8 | get_lines 1216–1228 | read other lines |
| — | set_timestamp never called | step limit hit, no start timestamp |
Prompt engineering did not fix this. We tested different wordings of the instructions and added few-shot examples of complete searches, and the model still kept searching after it had found the matching line.
Building a Training Set with a Custom Action Recorder
Because prompting alone did not produce reliable results, we decided to fine-tune a model on examples of the task done correctly, so that it would learn to stop searching and commit the timestamp once a search had found the matching line. To create the fine-tuning dataset, we built an annotation tool. It had all of the agent's tools available, and we invoked them by hand as we worked through finding real target sentences in real transcripts, while the tool recorded every action we took as a structured trajectory. Each trajectory was the ordered list of tool calls we made for one target, with the arguments, our reasoning, and the result returned by each call.
The tool exposed the same six tools we had given the agent, for searching the transcript for a pattern, reading a specific range of lines, pulling more context from the target statement, setting a start or end timestamp, reading the start timestamp already set, and marking the turn as done. The first version of the tool, below, had a form for each of them.
For each statement, we need two timestamps, one for where it starts and one for where it ends. To find the start, we look for the statement's first three sentences in the Congressional Record. To find the end, we look for its last three sentences. Once we have found both, we take them as the statement's start and end timestamps and cut the audio and video between them.
The core of the recording is the search. We take a short, distinctive phrase from the target statement and search the transcript for it. Each match comes back as a window, which is the matched sentence plus the six transcript lines above and below it, each with its line number and its start and end time in milliseconds. The point of the search is to get a window of the transcript around the start or end of the statement. A model can then read that window, find the line where the statement starts or ends, and copy that line's start or end time, in milliseconds, as the statement's timestamp. While recording, we made that choice by hand, and those choices are what the model learns from.
In the example below, we are looking for the end of a statement by Rep. Katherine Clark on February 2, 2023. Her statement ends: "The American people don't see themselves in the Republican agenda, and I would ask the majority leader, what does he say to them? Thank you, and I yield back."
In this case, we would search for a snippet like "see themselves in the Republican agenda." It is distinctive: a phrase like this should appear only in her statement, so the search returns a single window. It also leaves off the start of the sentence, "The American people don't," because of the contraction. A contraction can appear differently in the two sources, so we generally avoided contractions in our searches. (In later versions of our search algorithm, we normalized contractions). "Thank you, and I yield back" would be a poor search: members say some version of it in routinely, so it would return windows from many speakers.
The table below shows the window the search returned: the lines around the end of her statement, as they appear in the transcript, each with its start and end time.
11524166.
| Line | Start (ms) | End (ms) | Transcript line |
|---|---|---|---|
| 1632 | 11456889 | 11468279 | The first bill of the 118th Congress was a bill that helps billionaires dodge their taxes and added $114 billion to the deficit. |
| 1633 | 11472362 | 11486978 | They continued their assault on reproductive freedom and are threatening economic disaster in order to cut Social Security and Medicare, and filling our schedule with hollow symbolic stunts. |
| 1634 | 11487779 | 11491762 | The American people are in the GOP's rearview mirror. |
| 1635 | 11492403 | 11495406 | It's politics over people, plain and simple. |
| 1636 | 11496006 | 11500250 | And our constituents and the American people are seeing this. |
| 1637 | 11500730 | 11511979 | A recent national poll found that 73% of Americans say House Republicans haven't paid enough attention to the country's most important problems. |
| 1638 | 11512660 | 11518043 | The American people don't see themselves in the Republican agenda. |
| 1639 | 11518583 | 11522465 | And I would ask the majority leader, what does he say to them? |
| 1640 | 11523165 | 11524166 | Thank you, and I yield back.end line |
| 1641 | 11524786 | 11526787 | Thank the gentlelady for yielding back. |
| 1642 | 11528748 | 11540118 | Say to the American people is thank you for giving the Republicans the House majority to finally stop this mad rush towards socialism that we see in the last two years by the Biden administration. |
| 1643 | 11540438 | 11547725 | The taxing, the spending, the out of control policies that have led our country into one of the worst economic times we've ever seen. |
| 1644 | 11548145 | 11552890 | Inflation through the roof to the point where families can't even afford to put gasoline in their car. |
We recorded the start and the end of each statement separately. Finding one of them, say the end, takes a short sequence of actions. We search the transcript for a distinctive phrase from the target, as described above. If the search returns no results, we search for a different distinctive phrase. When a search returns one or more results, we inspect them by hand to find the last sentence of the speaker's statement, as given in the target. We record its end timestamp in milliseconds and then mark the turn as done. Finding the start works the same way, with the first sentence and its start timestamp. The goal was to teach this manual procedure to a model so that it could repeat it on its own.
A script then turns each sequence into training data, one step per action. Each step pairs an input, the target statement plus every search made so far, with the action we took next, written as JSON. A sequence with one search, one timestamp, and a done therefore becomes three training steps, as in the example below.
"It also saves this country and gives my little Matthew, who is 6 months old, a future. That is the morality. Mr. Speaker, I yield back the balance of my time."Each row below is one recorded training step. The model receives the input and is trained to produce the action.
| Step | Input | Action |
|---|---|---|
| 1 | The target, quoted above. No searches yet. |
search_pattern "also\s+saves\s+this\s+country\s+and\s+gives\s+my\s+little"
A distinctive phrase from the target.
|
| 2 |
The target, plus the search result (one match, transcript lines 3393–3405), which includes:
3399 It also saves this country and gives my little Matthew, who's six months old, a future.
3400 And that's the morality. 3401 With that, I yield back. (ends 21844973) 3402 The gentleman yields back. |
set_timestamp end = 21844973
Line 3401 is the last sentence of his statement, and the next line is the presiding officer, so line 3401's end time is the end of the statement.
|
| 3 | The target, the search result, and the end timestamp 21844973. |
done
The end timestamp is set, so we mark the turn as done.
|
Search heuristics
While recording, we favored simple, robust search patterns rather than long verbatim quotes, because it was unclear how transcription would render certain text. The patterns we leaned on mostly avoided:
- Contractions: the transcript may expand or split them (
don'tvs.do not). - Proper nouns: names and places are frequently misspelled or transcribed inconsistently.
- Punctuation: commas, dashes, and quotes rarely line up between the statement and the transcript.
- Numbers and percentages: these may appear as digits or words (
73%vs.seventy-three percent).
At this stage, we simply searched carefully around these differences. Later versions of our search normalize contractions, punctuation, and numbers automatically.
Smoke Test: Fine-Tuning on a Small Dataset
After we had recorded about 80 steps across a dozen statements, we ran a quick smoke test to see whether the data could teach the task at all.
gpt-oss-20b
For the smoke test we used OpenAI's gpt-oss-20b, a strong open-weights model that we could fine-tune ourselves, on a single RTX 3090 with 24 GB of video RAM (VRAM). We fine-tuned it on all but the last three recorded statements and kept those three for evaluation. These three recorded statements had a total of 21 input-output steps. To evaluate a model, we gave it the input from each of those 21 steps and compared the action it produced with the action we had recorded. We ran this evaluation on the untrained model and on each fine-tuned version. The untrained model ran through llama.cpp's llama-server using GGUF weights (llama.cpp's model file format). Fine-tuning it ran into five failures.
Failure 1: Training crashed on Windows
We started on Windows, training with the Hugging Face Transformers library. Every training run crashed with CUDA illegal-memory-access errors and driver resets while the model was loading or early in training.
We tried two changes to reduce memory use. First, we shrank the adapter, the small set of extra weights that fine-tuning actually trains. Instead of adding adapter weights to every linear layer in the model, we added them only to the expert feed-forward weights in three of the model's 24 layers (layers 7, 15, and 23). Second, we told Transformers to load the entire model onto the GPU, rather than letting it decide on its own how to split the model between the GPU and the computer's main memory. Neither change helped. Training still crashed the same way, while loading the model or early in training.
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Switching from Windows to Ubuntu
The crashes left us with no working way to train on Windows, so we made a bigger change: we replaced Windows with Ubuntu on the training machine. We expected Linux to handle GPU memory better and to let us use tools like Unsloth that make local fine-tuning more workable. It meant rebuilding the whole setup on the new OS, including GPU drivers, CUDA, the Python environment and llama.cpp, before we could train again.
Failure 2: The model would not load for training
On Ubuntu, training with Transformers ran out of memory while loading the model. gpt-oss-20b ships in a compact 4-bit format (MXFP4), but the standard Transformers recipe for fine-tuning it first expands the weights to 16-bit (BF16), and at 16 bits a 20-billion-parameter model does not fit in 24 GB. We switched training to Unsloth, a fine-tuning library that keeps the model in 4-bit during training, and training worked.
Loading the fine-tuned model back into Transformers to test it also ran out of memory: Transformers can only run the 4-bit format directly with Hugging Face's kernels package installed, which we didn't have, so it expanded the model to 16-bit.
We tested through Unsloth instead, loading the 4-bit base model with the trained adapter. That ran, but at about 8 tokens per second, against about 50 for the untrained model on llama-server, and some evaluation steps still ran out of memory.
So we converted each fine-tuned model to GGUF and tested it through llama-server, the same way we ran the untrained model. That fixed the memory problem: the fine-tuned model ran at about 55 tokens per second with no out-of-memory errors. But only 11 of its 21 evaluation responses were valid JSON.
Failure 3: Broken output format
Each action is a standard LLM tool call, a short JSON record that names the tool and gives its inputs. The search tool has one main input, the phrase to search for, and two optional ones, how many lines to show above and below each match (before and after, 6 by default). We also had the model write out its reasoning with every action. Our recorder, however, saved the inputs of every tool with every action. So each search in the training data also carried the inputs of the other tools, such as the start_line and end_line of the line-reading tool and the field_name and value of the timestamp tool, all left empty (null). Below is the search from the example above, first as it was stored in the training data and then with only the search tool's inputs and the reasoning.
{"action": "search_pattern", "pattern": "also\\s+saves\\s+this\\s+country\\s+and\\s+gives\\s+my\\s+little", "before": 6, "after": 6, "direction": null, "sentence_count": null, "start_line": null, "end_line": null, "field_name": null, "value": null, "reasoning": "Search distinctive anchors that should survive transcription drift."}
{"action": "search_pattern", "pattern": "also\\s+saves\\s+this\\s+country\\s+and\\s+gives\\s+my\\s+little", "before": 6, "after": 6, "reasoning": "Search distinctive anchors that should survive transcription drift."}
The fine-tuned model learned to copy those empty fields, and then it got them wrong. In the output below, it starts a normal search, but several field names lose their quotation marks or disappear, so the record is no longer valid JSON and our code cannot run the search. It then keeps writing past the end of its answer. It writes the special markers gpt-oss uses to close one reply and open another (<|end|><|start|>assistant), and in that new reply it talks to itself about its JSON being cut off and says it will try again. Our code cannot use any of that either.
{"action": "search_pattern", "pattern": "FBI pays Twitter \\$3 million", "before": 3, "after": 3, "direction": null, "sentence_count": null, "start_line\": null, "
: null, "field_name\": null, "
: null, "reasoning\": \"I will search for a distinctive phrase that will not be missed in fuzzy search. If I do not get a result I will simply choose another search phrase.\"}<|end|><|start|>assistant<|message|>I see that the action was not fully returned. It seems my JSON string was cut off. I should try again, ensuring I provide a complete JSON object. Perhaps I should use shorter keys. But the allowed tools specify the keys. I must use them. I will try again with a simpler string. [...] Let's try."
: 2
}
We already had llama-server's structured output setting turned on, with a JSON schema for the tool call (response_format), which is supposed to force the output into that format. That did not stop this model from garbling the field names. Turning the schema off was worse, because the server could not parse the model's output at all and returned an error. Lowering the model's reasoning effort from medium to low did not help either.
To check whether the server was at fault, we ran the same evaluation on the untrained model. Both models got the same input for each of the 21 evaluation steps, the instructions, the target statement, and the searches so far, through the same server with the same schema setting. Each time, we expected one tool call back, like the search shown above. The untrained model returned a valid tool call on all 21 steps. The fine-tuned model returned valid JSON on only 11, and only 3 of those were tool calls our code could run. So the garbled field names were something the fine-tuned model had learned, not a problem with the server.
To fix it, we changed the recorder to save only the inputs each tool actually uses, cleaned the existing recordings the same way, and fine-tuned the model again. The new model's field names came out correctly, with the same schema still in place. The schema had not fixed the problem, but cleaning the training data did. In our best fine-tuned version, 19 of the 21 evaluation responses were valid tool calls with the correct field names. The other 2 came back empty, which is the next failure we discuss.
Failure 4: Reasoning used up the token budget
gpt-oss-20b reasons before it answers. It writes its reasoning first and its answer second, and llama-server returns the two separately. Every request also sets a maximum response length, which we normally set to 1,024 tokens. If the model is still reasoning when it reaches that length, it is cut off before it writes an answer, and the answer comes back empty.
We first ran into this by accident. While debugging, we lowered the maximum to 128 tokens, and none of the 21 evaluation responses came back valid. In the response below, the reasoning stops mid-sentence and the answer is empty.
"finish_reason": "length",
"content": "",
"reasoning_content": "I need to find the start of a statement in a long transcript. The core rules say I should do fuzzy matching and not require exact wording. [...] I need to choose a tool. The most direct",
"usage": {"completion_tokens": 128, "prompt_tokens": 592}
Restoring the 1,024-token maximum fixed most of these, and the next run returned valid JSON on 18 of the 21 steps. Later fine-tuned versions, however, came back empty on 9 of the 21 steps in one run and 15 in another. To rule out the server, we ran the untrained model again with the same settings, and all 21 of its responses were valid.
The cause was the same. With 1,024 tokens, the fine-tuned models sometimes spent the entire response reasoning, which is why each empty response took about as long as writing 1,024 tokens. We tried raising the maximum to 1,536 tokens, lowering the model's reasoning level from medium to low, and asking it in the prompt to keep its reasoning brief. A longer maximum, though, also needs more GPU memory, and on a single 3090 memory was already a constraint.
The table below shows one example. For one statement in the evaluation set, we had made five tool calls while finding its start: three searches, then setting the timestamp, then done. To evaluate the fine-tuned model, we replayed that recording one tool call at a time. At each point, we gave the model everything we had seen up to then and asked for its next tool call. The table compares its answer with ours at each of the five points. Twice it returned nothing, including at point 4, where our earlier searches had already found the start and the correct tool call was to set the timestamp.
| Point | Our tool call | Model's tool call |
|---|---|---|
| 1 | search_pattern | Empty response |
| 2 | search_pattern | search_pattern |
| 3 | search_pattern | search_pattern |
| 4 | set_timestamp | Empty response |
| 5 | done | done |
Failure 5: Fine-tuning lowered accuracy
Our baseline was gpt-oss-20b as it comes out of the box, with no fine-tuning. The goal of fine-tuning was to start from that baseline and raise its accuracy to near 100%. Out of the box, the model already returned a valid tool call on all 21 evaluation steps and picked the correct tool (search, set a timestamp, or done) on 16 of them. Instead of improving on that, fine-tuning made it worse. The best fine-tuned version returned a valid tool call on 19 steps and picked the correct tool on 14.
Fine-tuning also did not teach the model to stop searching, which was the main reason we fine-tuned. Even when a search result contained the matching line, the fine-tuned versions often did not set the timestamp and searched again for another phrase from the target.
Gemini 2.5 Flash
To get around the local problems, we fine-tuned a larger hosted model, Gemini 2.5 Flash, on Google's Vertex AI. We used the same 66 training steps and evaluated it on the same 21 steps. It had none of the local failures. Every response came back as a valid tool call, and it picked the correct tool on 20 of the 21 evaluation steps. Its one wrong tool was setting the timestamp a second time when it should have marked the turn as done.
Most important, it stopped searching when it should. At all 6 evaluation steps where the correct tool call was to set the timestamp, it set one instead of searching again. It returned the correct timestamp on 3 of those 6. On the other 3 it returned the timestamp of a neighboring transcript line.
Its searches were weaker. It rarely used the kind of short phrase we recorded and often searched for a whole sentence copied from the target, punctuation included, such as "Mr. Speaker, how 'bout them Dawgs?" A search like that is less likely to match the transcript.
Smoke test conclusions
The results for the local model were inconclusive. With only 66 training steps, we could not tell which failures came from the data, which came from the model, and which came from running it on a local GPU. Still, manual inspection of the model outputs showed that the fine-tuned models were clearly learning from our data. Even when their output was not valid JSON, it used the same tool names and fields as our recorded actions, and repeated our reasoning sentences nearly word for word. That suggested more data might help fix many of the problems we encountered.
Gemini showed that a model fine-tuned on our recordings could learn the task.
Together, these results gave us confidence to keep going. If a local model did not perform as well, we could try to improve it with more and better data or try a different local model. If all else failed locally, a cheap, well-trained hosted model might still be an option for production. So we went back to recording.
Recording the Full Dataset
Readers may remember that the annotation tool we built offered six tools to help find and commit start and end timestamps:
search_pattern, to search the transcript for a phrase (regular expressions allowed)get_lines, to read a specific range of transcript linesget_target_context, to pull more of the statement than the three sentences provided by default (its first three when finding the start, its last three when finding the end)set_timestamp, to set the start or end timestamp, given in milliseconds (only a timestamp that matches the start or end time of a transcript line is accepted)get_start_boundary, to read the start timestamp already set, used when finding the end timestamp to rule out matches before the startdone, to mark the turn as done
However, while using our annotation tool to record the initial smoke-test dataset, we found that in practice three tools were enough to find every start and end timestamp: search_pattern, set_timestamp, and done. Based on this, we reduced the toolset in the fine-tuning dataset to those three tools.
We also rebuilt the annotation tool around those three tools. Below is a screenshot of the updated version. The statement we are recording is on the left, with its start and end chunks, and the search tool is underneath it. On the right are the only two other tools, Set Timestamp and Done, and a running list of the actions recorded so far.
A closer view of the top of the tool shows the selected statement next to the Set Timestamp and Done forms.
Search results now list each transcript line with its start and end time, and clicking a time fills in the timestamp form. Below is one search result from the recorder.
We kept recording until we had about 1,200 steps from ten floor sessions (House and Senate days from early 2023, plus one later Senate session), split roughly evenly between start boundaries (617) and end boundaries (580). We judged that to be enough to tell us whether this fine-tuning approach would work.
We aimed to record a variety of statements to get a good, diverse sample. Because the task is only to find the start and end lines, the variety we needed was in how statements begin and end on the floor. Some open by thanking "Mr. Speaker," "Madam Speaker," or "Mr. President," and others skip that entirely. Some members yield themselves time, interrupt another speaker, or are interrupted. Some statements end with a formal "I yield back," and others simply stop. We checked the mix of these cases qualitatively as we recorded.
Fine-Tuning on the Full Dataset
For the full dataset, we again fine-tuned one hosted model and one local model. On the hosted side we kept Gemini 2.5 Flash, which had done well in the smoke test, and also fine-tuned Gemini 2.5 Pro to see whether a larger, more powerful model helped. On the local side we switched from gpt-oss-20b to Qwen3-14B, which benchmarks show is a strong open model with good tool-use behavior. It is also smaller than gpt-oss-20b, and Unsloth supports fine-tuning it in 4-bit on a 24 GB card, which mattered after the memory problems in the smoke test. A local model was still the preferred outcome. A model we run ourselves is essentially free, and once API latency is factored in, its speed is competitive with a hosted model.
For each statement, the fine-tuned model is given the target text and the same three tools discussed in the previous sections: transcript search, timestamp setting (start or end boundary), and a tool to mark the run as done. To evaluate the model, we checked whether the timestamp it set matched the true start or end of the target statement in the transcript. To evaluate the model further, we read through selected model tool calls to see how it actually behaved.
Qwen3-14B
We trained and ran Qwen3-14B with Unsloth and the model loaded in 4-bit (QLoRA). The main constraint was context. The model's input for each tool call includes the target statement and the full tool history up until that point (every search already tried and the window it returned), so the input gets longer with every tool call. Our first Qwen run capped each training example and each input at 2,048 tokens.
We trained Qwen3-14B on the full ~1,200 steps and evaluated it on a House session from February 2, 2023, which we held out of training entirely. That session had 109 statements, so the evaluation set had 109 start timestamps and 109 end timestamps to find. Qwen3-14B did not perform at a level that would meet production quality. We ran into three failures, each explained in detail below.
Failure 1: Context overflow at 2,048 tokens
In the first run, the input grew with every tool call, and after one or two tool calls it was longer than the 2,048-token cap. The training data had the same problem, and 58 of the 1,197 examples had to be cut to fit.
An input longer than the cap was cut off at 2,048 tokens before it reached the model, often in the middle of a transcript line. The cut also removed the end of the input, where Qwen's chat format marks the start of the model's turn and closes its reasoning block. So instead of answering, the model first finished the cut-off line, as if it were still reading its input, then wrote </think> to close the reasoning block, and only then wrote the JSON tool call. In the raw output below, from the second tool call of the end run, the input was cut partway through transcript line 1723, right after start_ms: 1. The model's response starts by finishing that line, 1973802 | end_ms: 11989711 | line#: 1723, before it writes </think> and the JSON.
1973802 | end_ms: 11989711 | line#: 1723
</think>
{"action": "set_timestamp","field_name": "end","value": 11524166,"reasoning": "The search result matches the target statement. I will set the end timestamp to the last sentence of the speaker's statement."}
As part of our evaluation, we ran the model on the same statement as the search example earlier, by Rep. Katherine Clark, first to find its start and then to find its end. Finding the start took 3 tool calls. Finding the end should also have taken 3, a search, setting the timestamp, and done, but the model got stuck. It searched for "I would ask the majority leader," correctly set the end timestamp from the returned window (lines 1633–1645), and then, instead of calling done, re-issued the identical set_timestamp six more times until the run hit its eight-step cap. Without a done signal, in production the timestamp would never be set and no media could be cut for the statement. The table below shows that run.
| Step | Action | Result |
|---|---|---|
| 1 | search_pattern "I\s+would\s+ask\s+the\s+majority\s+leader" | matched correct line |
| 2 | set_timestamp end = 11524166 | timestamp set |
| 3–8 | set_timestamp end = 11524166 | repeated |
| — | "done" never called | step cap hit |
Across both runs, the model made 11 tool calls. Because each tool call adds to the input for the next, the first one or two calls in each run had inputs under the 2,048-token cap, and every later call, with more tool history in its input, was over it, 8 of the 11 in all. All 8 came back with the broken output described above, and the 3 under the cap came back clean.
For the second run we compacted the tool history. Instead of repeating every earlier search result in full, each one was summarized as the phrase searched plus the matched line and the first and last lines of the window it returned. That cut the typical training example from about 1,300 tokens to about 800, and the longest from about 12,500 to about 3,700. With shorter examples, we raised the cap to 4,096 tokens, and no example had to be cut. With the compacted history and the higher cap, the same end of Clark's statement went search, set, done. The boxes below show one training step, the Schweikert example from earlier, before and after compaction.
TOOL HISTORY:
Step 1 action={
"action": "search_pattern",
"pattern": "also\\s+saves\\s+this\\s+country\\s+and\\s+gives\\s+my\\s+little",
"before": 6,
"after": 6,
"reasoning": "Search distinctive anchors that should survive transcription drift."
}
Step 1 result:
Found 1 merged window(s) for pattern `also\s+saves\s+this\s+country\s+and\s+gives\s+my\s+little`. Returning all merged windows (overlapping raw hits are collapsed).
Field note: duplicate_window_count is the number of returned windows in this search whose full normalized window text is identical to this one.
Field note: overlapping raw match windows are collapsed into a single merged window, so returned windows may be larger than the requested before/after context.
MATCH 1 | matched_line=3399 | window=3393-3405 | duplicate_window_count=1
It's not the shiny object that gets you applauded when you go into your town hall meeting. | start_ms: 21813311 | end_ms: 21818034 | line#: 3393
Oh, we did this. | start_ms: 21818694 | end_ms: 21819374 | line#: 3394
This stuff is hard. | start_ms: 21820015 | end_ms: 21820995 | line#: 3395
It's complicated. | start_ms: 21821496 | end_ms: 21822316 | line#: 3396
You are going to be lobbied like a war. | start_ms: 21822576 | end_ms: 21824757 | line#: 3397
They're going to spend money in your district beating the crap out of you because you're taking away their money. | start_ms: 21825178 | end_ms: 21829640 | line#: 3398
It also saves this country and gives my little Matthew, who's six months old, a future. | start_ms: 21832788 | end_ms: 21839771 | line#: 3399
And that's the morality. | start_ms: 21842592 | end_ms: 21843773 | line#: 3400
With that, I yield back. | start_ms: 21844093 | end_ms: 21844973 | line#: 3401
The gentleman yields back. | start_ms: 21844993 | end_ms: 21848495 | line#: 3402
The gentleman yields back. | start_ms: 21848515 | end_ms: 21850956 | line#: 3403
The gentleman yields back. | start_ms: 21850976 | end_ms: 21851636 | line#: 3404
The gentleman yields back. | start_ms: 21851656 | end_ms: 21852516 | line#: 3405
TOOL HISTORY:
Step 1 action={
"reasoning": "Search distinctive anchors that should survive transcription drift.",
"action": "search_pattern",
"pattern": "also\\s+saves\\s+this\\s+country\\s+and\\s+gives\\s+my\\s+little",
"before": 6,
"after": 6
}
Step 1 result:
result_summary: 1 match_windows; timestamp_set=no
match_windows:
- match 1: matched_line=3399; window_lines=3393-3405; duplicate_window_count=1
matched_line_text: line 3399 start_ms=21832788 end_ms=21839771 text="It also saves this country and gives my little Matthew, who's six months old, a future."
window_start_text: line 3393 start_ms=21813311 end_ms=21818034 text="It's not the shiny object that gets you applauded when you go into your town hall meeting."
window_end_text: line 3405 start_ms=21851656 end_ms=21852516 text="The gentleman yields back."
Failure 2: Reasoning that did not guide the action
Every recorded action included our written reasoning, and we trained the model to write reasoning too. In the first Qwen format, the reasoning came at the end of each action, after the tool and its inputs. A model writes its output in order, so it had already chosen the action before writing any reasoning, and the reasoning could not affect the choice.
The reasoning it wrote was also mostly copied from the training data. Our recorded reasoning reused the same few sentences over and over. The explanation for a first search appears word for word in 246 of the 1,197 training steps, and Qwen wrote that exact sentence for every first search.
For the second run, we moved the reasoning in each recorded action ahead of the tool call, following the common advice to have a model think through a problem before it answers (see, for example, Anthropic's prompting guide). With the reasoning first, the model can use it when choosing the tool call and its parameters, and every tool call in the training data is preceded by an explanation of why it was chosen.
The output followed the new order, but the reasoning was still the same copied sentences. The boxes below show the model setting the start timestamp of Clark's statement in each run. The reasoning moved to the front, but it is the same sentence both times, and neither timestamp is correct. The first is from a different member's speech, about 92 minutes before Clark's statement, and the second does not exist anywhere in the transcript (Failure 3).
{"action": "set_timestamp","field_name": "start","value": 5875939,"reasoning": "The search result matches the target statement. I will set the start timestamp to the first sentence of the speaker's statement."}
{"reasoning": "The search result matches the target statement. I will set the start timestamp to the first sentence of the speaker's statement.","action": "set_timestamp","field_name": "start","value": 11376787}
Failure 3: A bug in the compaction
We evaluated the model trained with the compacted history on Clark's statement again, but then discovered that our compaction had a bug. It was meant to shorten only the earlier search results, but it also compacted the most recent one, which is the result the model reads to choose the timestamp. So at the step where it set a timestamp, the model saw only the matched line and the first and last lines of the window. In the compacted training data, the correct timestamp appeared in the model's input for only 74 of the 312 set_timestamp examples. For the other 238, the model was trained to produce a timestamp it could not see. In the Schweikert example above, the correct end timestamp, 21844973, appears before compaction but not after.
In evaluation, the model invented a timestamp in exactly this situation. For the start of Clark's statement, its search returned a window containing the true start, line 1623 ("I thank the gentlemen, and it is my privilege to join my first colloquy…", 11376016), but its input for the next tool call showed only the matched line and the first and last lines of that window. The model set 11376787, a value that does not exist anywhere in the transcript. set_timestamp only accepts a timestamp that matches the start or end time of a transcript line, so it returned an error, and the model called done, leaving the start unset.
| Step | Action | Result |
|---|---|---|
| 1 | search_pattern "a\s+very\s+local\s+flavor\s+to\s+it" | window 1618–1630; next prompt shows lines 1618, 1624, 1630 only |
| 2 | set_timestamp start = 11376787 | not a transcript timestamp |
| 3 | done | start never set |
We found the bug and built a corrected training set that kept the most recent search result in full, but we never trained on it. Instead, we decided to step back and rethink the problem more fundamentally, as described in Rethinking the Task below.
Gemini 2.5 Flash and Pro
While we were training Qwen3-14B, we also fine-tuned Gemini 2.5 Flash and Pro on the same recorded steps and tested them on the same House session. They had three failures of their own, each explained in detail below.
Failure 1: Truncated JSON output
The most frequent failure was invalid output. We wanted structured tool calls, and the standard way to request structured output from Gemini on Google Vertex is to set response_mime_type="application/json". But for these tuned endpoints, that did not make the outputs reliable. The models repeatedly returned truncated JSON—unexpected given our previous experience with Gemini, and a serious problem for a production deployment.
| Model | What happened | Result |
|---|---|---|
| Gemini 2.5 Flash | Start-timestamp response broke off mid-string | no action executed |
| Gemini 2.5 Pro | Same prompt truncated with JSON mode enabled | invalid JSON |
| Gemini 2.5 Pro | Same prompt with JSON mode disabled | valid tool action |
We attempted to debug the issue by saving the full raw model response whenever the output could not be parsed, and we limited how many tokens the model could return so a single action could not run on too long. This did not provide broader insight. We then reexamined the documentation and noticed that Google Vertex's supervised tuning documentation warns that controlled generation can reduce quality for tuned Gemini models because the tuning job does not apply controlled generation during training. That matched what we saw with Gemini 2.5 Flash and Gemini 2.5 Pro: JSON mode made malformed or truncated output more likely. Disabling JSON mode reduced the output-format problem, but not completely, and other error states remained (see Failure 2 below).
Reformatting and more training didn't fix it
We suspected the training format was part of the problem, so we retrained Gemini 2.5 Flash on the same rebuilt data we used for the final Qwen3-14B run: the human-written reasoning placed before each action, in the hope it would better guide the model's choice, and a compacted tool history. We also trained longer, with five epochs instead of one and a larger adapter (size 8 instead of 4). Finally, we fine-tuned Gemini 2.5 Pro on the same data to test a stronger model. None of these changes made a substantial difference in performance.
Failure 2: Brittle search, boundary set too early
Gemini 2.5 Flash's search choices were often brittle. As described above, our training data favored simple, robust search patterns—short phrases that mostly avoid contractions, proper nouns, punctuation, and numbers or percentages, since it is unclear how the transcript will render those. Gemini 2.5 Flash often did the opposite and searched for a whole sentence copied from the statement. When looking for the end of Clark's statement, its first three searches were each a full sentence from the Congressional Record, and none of them matched. The transcript splits the first and third sentences across two lines with different punctuation, and it writes the "73 percent" in the second sentence as "73%." Its fourth search, "politics over people, plain and simple," matched line 1635. It then set the end timestamp to the end of line 1637, three lines before the true end on line 1640.
| Step | Action | Result |
|---|---|---|
| 1 | search_pattern "politics over people, plain and simple, and our constituents and the American people are seeing this." | no match |
| 2 | search_pattern "A recent national poll found that 73 percent of Americans say House Republicans haven't paid enough attention to the country's most important problems." | no match |
| 3 | search_pattern "The American people don't see themselves in the Republican agenda, and I would ask the majority leader, what does he say to them?" | no match |
| 4 | search_pattern "politics over people, plain and simple" | matched (shortened) |
| 5 | set_timestamp end → line 1637 | set too early |
Failure 3: Gemini 2.5 Pro gave up on a present statement
Gemini 2.5 Pro made the output-format issue easier to diagnose, but it did not solve the task. With JSON mode disabled, Pro was able to emit more valid tool calls, but on the February 2 House evaluation it matched only 1 of 196 timestamps exactly. Most runs either kept searching until the step limit, decided the statement was not in the transcript, or still returned malformed output.
| Step | Action | Result |
|---|---|---|
| 1–6 | search_pattern variants from Katherine Clark's opening | no match |
| 7 | done — "statement is not in the transcript" | missed timestamp |
| truth | start timestamp = 11376016 | line exists |
After these runs, neither the local nor the hosted models were close to usable, and more data had not clearly helped. The 1,200 recorded steps did no better than the 80 in the smoke test. We weighed training longer, trying a different model, or changing the approach, and chose to change the approach, as described next.
Rethinking the Task
Stepping back, we weighed several reframings of the problem. One was to treat the timestamp step as a classification problem—training a dedicated classifier to pick the correct line from a set of candidates. We never ended up shipping a classifier, but reasoning in those terms led us to consider splitting the task.
The idea was to split the task into two separate jobs, one that only searches and one that only picks the timestamp, and to test whether the simpler tasks helped. So we split the work into two narrower jobs and benchmarked each on its own.
The Idea: Decompose Into Two Specialists
A person doing this task does two things. First, they pick a distinctive phrase from the target statement and search the transcript for it. Then they read the lines the search returns and find the line where the statement starts or ends. We turned each of these into a separate task for a model.
- Search-policy model — picks a phrase from the target statement to search the transcript for. If an earlier search found nothing, it picks a different phrase.
- Timestamp-identification model — given the target sentences and the transcript lines a search returned, selects the line where the statement starts or ends, or says that none of them is correct.
The two tasks run one after the other. The search model picks a phrase, our code runs the search, and the timestamp model selects a line from the results. If it says none of the lines is correct, the search model picks a new phrase.
Splitting the task also keeps each model's input short. The search model sees the phrases already tried and a short summary of what each search returned, such as the number of matches, but not the transcript lines themselves. The timestamp model sees only the result of the latest search, in full. Unlike our earlier compacted history, nothing the model needs for its current decision is left out, and neither input grows much as more searches are run.
A Labeling Trick
Thinking of the timestamp step as classification also made us ask whether we could simplify it further. Until then, the model set a timestamp by copying its millisecond value from the search results, which meant writing out an exact eight-digit number. Choosing one option from a short list is a simpler task, so we stopped showing the model timestamps and asked it to choose a line instead. This also made the input marginally shorter, since previously every line in a search result had listed both its start and end time.
When a search returns results, we give every line in them a short label. As a reminder, each search result is the transcript line that matched plus the six lines above and below it, forming a window of 13 lines in total. M stands for match window. M1 is the first match window for the search phrase, M2 the second, and so on. The number after the dash is the line's position in that search result. M1-6, for example, is the sixth line of the first search result (i.e. the first match window). Each line has one label, whether we are looking for the start or the end of the statement. The timestamp model returns one of these labels, or none if it cannot find a line it thinks matches the target. When we label the lines, we also keep a table that maps each label back to its line's start and end timestamps.
The search result below is from a search for the start of a statement by Rep. Raúl Grijalva, which begins "Mr. Speaker, I thank the gentleman for the time. As Ranking Member Nadler from the Judiciary Committee said…"
MATCH 1 | matched_line=1469 | window=1463-1475
[M1-1] The bottom line is, you know, this committee is basically an assurance
[M1-2] That we're going to see these very partisan, political, MAGA-driven investigations go forward.
[M1-3] And again, I urge my colleagues on both sides of the aisle to vote no.
[M1-4] I want to now yield two minutes to the gentleman from Arizona, Mr. Grijalva.
[M1-5] The gentleman from Arizona is recognized for two minutes.
[M1-6] Thank you, and thank you, Mr. McGovern, for the time.
[M1-7] You know, as Ranking Member Nadler from Judiciary said, H.R.
[M1-8] 12 Is a dangerous government attempt to interject extremist politics into our justice system and shield MAGA extremists from legal consequences for their actions, including January 6th, and the follow-up and the investigations that are ongoing now.
[M1-9] He's right about...
[M1-10] About that Republican strategy.
[M1-11] H.R.
[M1-12] 12 Is both political cover and a political weapon.
[M1-13] Like any legislation, HR 12 is driven by the motivation of its sponsors, the Republican majority in the House of Representatives.
The model should return the label M1-6, the highlighted line. Because we are looking for the start of the statement, our code would look up M1-6 in the table and use the start time of line 1468, which is 9025361 milliseconds. A label that is not in the table is rejected, so the model cannot produce a timestamp that does not exist.
Stage One: The Search-Policy Model
The search-policy model has one job. Given the target sentences and the searches already tried, it returns a new phrase from the target to search the transcript for.
Two design choices mattered here. First, we ask for literal snippets, not regex. In our experience the regular-expression flexibility the old agent had was never needed. A short distinctive phrase copied from the target works better and is far easier to reason about. Second, the model is shown the searches that already failed, so it can avoid searching for duplicate phrases and pivot to a new search phrase.
The box below shows the fine-tuned Gemini 2.5 Pro's first search when looking for the start of Rep. Katherine Clark's statement, whose target sentences end "I really do appreciate the insight into the week ahead, although it does seem to have a very local flavor to it." The model searched for the second half of that sentence, which matched only Clark's statement. The before and after fields set how many lines are returned above and below the match. They default to 6, and in practice six lines on each side was enough, so we never needed to change them.
{
"action": "search_pattern",
"pattern": "although it does seem to have a very local flavor to it",
"before": 6,
"after": 6,
"reasoning": "I will choose a distinctive phrase from the target statement that is likely to be transcribed plainly, avoiding bill titles, proper nouns, quotes, acronyms, and punctuation-dependent text."
}
We fine-tuned Qwen3-14B, Gemini 2.5 Flash, and Gemini 2.5 Pro for this task on the 541 search steps from our recorded data. Each step pairs the target sentences and any earlier searches with the phrase we searched for next. The model searches for one phrase after another until a search finds a match.
A timestamp has one correct value, so it can be scored as right or wrong. A search phrase cannot, because many different phrases from the target could each find the statement. So we built an automated scorer that flags measurable problems rather than demanding an exact match: searching for text outside the target, leaning on an over-generic phrase, repeating a failed query, or fragile capitalization assumptions.
We ran the scorer on all three fine-tuned search models. Each was tested on 20 statements that were not in the training data, searching for both the start and the end of each, for 40 cases with up to six searches per case. The table below shows the results.
| Measure | Qwen3-14B | Gemini 2.5 Flash | Gemini 2.5 Pro |
|---|---|---|---|
| Cases where a search found a match | 37 of 40 | 36 of 40 | 39 of 40 |
| Found on the first search | 26 | 28 | 31 |
| Total searches | 69 | 75 | 57 |
| Repeated a failed search, exactly or nearly | 7 | 25 | 0 |
| Capitalized first letter | 10 | 24 | 6 |
| Overly long, specific phrase | 7 | 13 | 7 |
| Too short or generic phrase | 8 | 6 | 4 |
| Phrase not in the target sentences | 14 | 5 | 2 |
Gemini 2.5 Pro found a match in more cases, needed fewer searches, and never repeated a failed search, so we chose it as the search model. Pro costs more per token than Flash, but each search uses only a few hundred tokens of input and a short output, so the difference in cost was small. Across our pipeline, we generally choose accuracy over cost.
In Retrospect, Training on Outcomes
There was another way to train the search model that we did not consider at the time. Instead of learning to copy the phrases we chose, a model can learn from outcomes. It tries many phrases for each statement, each search is checked for whether its results contain the correct line, and the phrases that do are rewarded. This also solves the scoring problem above, because a phrase is judged only by whether it leads to the correct line.
We had the data to try it. Training this way needs statements with known correct timestamps rather than more recorded steps, and every statement we recorded already had them.
In the end it was not needed. We kept simplifying the approach, and the automatic search described below found the correct line for every timestamp in our benchmark without any training.
Stage Two: The Timestamp-Identification Model
Once a search returns results, the timestamp-identification model takes the target sentences and the labeled lines as input. It returns the label of the line it judges to be the first sentence of the target, or the last sentence when we are looking for the end. The transcript and the Congressional Record rarely match word for word, as described in A Reminder of Why Timestamp Matching Is Hard, so this is a judgment call rather than an exact match.
The model can also return none, for when none of the lines is correct.
We did not fine-tune this model. Instead, we gave existing models our prompt with a few worked examples and tested them on a simple 50-sample benchmark we constructed from our existing data. Each sample gives the model the target sentences and the labeled lines from one of our recorded searches, and asks it for the line where the statement starts or ends. We scored each answer against the line we had chosen by hand. The table below shows how many each model got right.
| Model | Correct line |
|---|---|
| Gemini 2.5 Flash, no thinking | 29 of 50 |
| Gemini 2.5 Pro | 41 of 50 |
| Gemini 3.1 Pro preview | 49 of 50 |
| GPT-5.2, medium reasoning | 32 of 50 |
| GPT-5.5, medium reasoning | 43 of 50 |
We also built a synthetic set of 50 samples to test whether a model correctly returns none when none of the lines in the search result match the target statement. Each sample pairs target sentences with the search result from a different statement. Gemini 2.5 Flash and Gemini 3.1 Pro returned none on all 50, and GPT-5.5 on 47. Gemini 3.1 Pro was the best model on both tests, picking the correct line on 49 of 50 and returning none on all 50 synthetic samples.
Replacing the Search Model with Automatic Search
We built the two models into our pipeline at the end of May, with the fine-tuned Gemini 2.5 Pro proposing search phrases and Gemini 3.1 Pro, the best model in our tests, picking the line. However, it was never run. We needed to shift our focus to the core parts of the pipeline and improve them, so that the target statements passed to media processing would be correct and complete. As discussed in our post Parsing the Congressional Record Ain't Easy, we had many bugs to work out.
When we came back to media processing in August, we realized the search did not need a model at all. The search model was following rules that code can apply directly. It took a distinctive phrase from the target, tried to avoid text the transcription might render differently, and tried a different phrase when a search found nothing. Code can do the same thing, and instead of trying one phrase at a time, it can search for every phrase in the target at once and keep the most distinctive matches. This changed our whole approach, and we replaced the search model with code that searches automatically.
This search is programmatic and deterministic. The code first normalizes the target sentences and the transcript in the same way. It lowercases the text, removes punctuation, expands contractions ("don't" becomes "do not"), and converts spelled-out numbers to digits and "%" to "percent." It also joins the transcript lines into one string, so a phrase can match even when the transcription splits it across two lines.
The code then breaks the target sentences into n-grams, every sequence of consecutive words from 12 words long down to 5. It does this with a sliding window, taking every 12-word sequence first, then every 11-word sequence, and so on. It searches the transcript for every n-gram. An n-gram with few matches is more distinctive than one with many, so the code ranks n-grams by their number of matches, fewest first, and breaks ties by length, longest first. It then takes up to three separate places where the top-ranked n-grams match, and each becomes a search result of the matched line and the eight lines above and below it.
We set the number of lines by testing windows from 2 to 30 lines on each side. Windows of fewer than six lines sometimes left out the correct line, and including windows of more than 8 lines on either side did not improve match performance.
The demo below runs this search on the end of Clark's statement, the same target Gemini 2.5 Flash failed to match in Failure 2. Click "Run" to see the animation.
Every 12-word n-gram from her first target sentence matches only once. Shorter n-grams such as "and I would ask the majority leader" match twice, because the same words appear again later in the transcript, so they rank lower. The search result for the top-ranked n-gram covers lines 1627 to 1643, the eight lines above and below where it matched, and contains the end of Clark's statement. The timestamp model then picks the line from this result.
To test the automatic search, we built a larger benchmark of 153 statements from 10 sessions, with 294 start and end timestamps that we labeled by hand.
Several newer models had come out since our first tests, so we also benchmarked the timestamp selection model again, giving each model the same prompt and the same search results. The table below shows how many of the 294 timestamps each got exactly right. It also shows the share within one line of the correct one. When cutting media we want the exact line, but when a model misses, being off by one line is generally acceptable, because the line gained or lost is usually filler at the start or end of the statement. The last column is the cost of running the whole benchmark.
| Model | Exactly right | Within one line | Cost |
|---|---|---|---|
| DeepSeek V4.1 Flash, high reasoning | 284 (96.6%) | 98.0% | $0.45 |
| DeepSeek V4.1 Flash, low reasoning (what we use) | 283 (96.3%) | 98.3% | $0.56 |
| Jev, passing to DeepSeek V4.1 Flash when under 70% confident | 283 (96.3%) | 97.3% | $0.34 |
| GPT-6 Luna Pro | 278 (94.6%) | 96.9% | $0.26 |
| GPT-6 Astra | 277 (94.2%) | 96.3% | $9.99 |
| GPT-6 Luna, medium reasoning | 277 (94.2%) | 96.6% | $0.15 |
| DeepSeek V4.1 Flash, medium reasoning | 277 (94.2%) | 96.9% | $0.29 |
| Gemini 3.8 Flash | 276 (93.9%) | 96.9% | $0.82 |
| GLM 5.3 | 273 (92.9%) | 94.9% | $0.60 |
| GPT-6 Luna, low reasoning | 271 (92.2%) | 95.6% | $0.12 |
| GPT-5.2 | 270 (91.8%) | 95.9% | $1.19 |
| GLM 5.3 Flash | 269 (91.5%) | 96.9% | $0.15 |
| Jev | 260 (88.4%) | 91.5% | $0.06 |
| GPT-5.4 mini | 239 (81.3%) | 91.2% | $0.60 |
We chose to use DeepSeek V4.1 Flash at low reasoning. It was among the most accurate models, getting 283 of the 294 timestamps exactly right, and running the whole benchmark cost about $0.56, a fraction of the cost of models with similar or lower accuracy, such as GPT-6 Astra at $9.99. At high reasoning it got one more timestamp right, but medium reasoning got six fewer, so the reasoning setting had no consistent effect and we treat a difference of one as noise.
Where We Are Now
Our current production setup combines exact matching with automatic search. Exact matching remains our first attempt at finding each statement's start and end timestamps. As a reminder, it searches the transcript for the statement's first or last sentence and, if there is no unique match, progressively shortens the sentence until there is one, as described in our post Timestamp Identification for Clip Cutting. Only when exact matching cannot place a timestamp do we fall back to automatic search, with DeepSeek V4.1 Flash picking the line.
After we parse the statements in the Congressional Record, we run this setup to find their timestamps and cut the audio and video for each statement. We are using it both for new sessions and to backfill media for previous years.
Looking back, each step made the task simpler. We started with a general-purpose agent with six tools, cut it to three tools and fine-tuned it, split the work into two separate jobs, and finally replaced the search model with code, leaving a single model call to pick a line. The final version is the simplest of them, and compared with the full-transcript fallback it replaced, it cuts costs by about 15x overall and 40x on the hardest statements and is about 40% faster.