A real verification need
People increasingly ask models “is this true?”. Tasks come from claims handled by professional fact-checking organizations, not synthetic Q&A.
The “No Tax on Tips” Savings Claim in Nevada
FACTCHECK-2350C1D7 · Open-web fact verification · tax policy claim
A politician claimed in an economic speech that “no tax on tips” saved 440,000 Nevada tipped workers an average of $10,000 last filing season. Within a limited search budget, the model must find evidence and decide whether the claim holds.
Fact-checkingOpen-web searchTax dataEnglish
5 / 8searches / page reads
50Kvisible characters at most
4scoring dimensions + 6 cap rules
3frontier models tested
< 65%frontier-model verdict accuracy
01
Highlights
The numbers must be taken apart
440,000 workers, $10,000, one filing season — each figure can hold or fail on its own, and the model must reach an overall verdict on the sentence.
Search decisions within a budget
At most 5 searches, 8 page reads and 50,000 visible characters per task. What to search and which page to read decide whether decisive evidence is found.
A cutoff against hindsight
Search results are filtered to before the cutoff date, so the model cannot copy the answer from later coverage.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 274 chars
Verify the public claim with the controlled search/read tools and submit a verdict with evidence.
Claim
Because of the 'no tax on tips' policy, 440,000 tipped workers in Nevada saved an average of $10,000 on their taxes in the past filing season.
Context
President Donald Trump made this claim during an economic speech at the Red Rock Casino Resort and Spa in Las Vegas on Aug. 5, 2026.
Evidence cutoff
2026-08-24. Search results are filtered to this date.
NOTEThis is the model's entire public input; tool definitions and the submission format are omitted.
03
Search environment
5searches (raw page text, no generated summaries)
8page reads
50Kvisible characters at most
2026.08.24evidence cutoff; later pages are filtered out
No attachments: all evidence comes from the model's own open-web searching and reading. The full tool-call record (queries, pages opened, passages cited) is saved with every attempt for traceability.
04
Scoring design
4scoring dimensions
+ 6 cap rules
Full criteria text, weights and reference answers ship with the complete data package.
4 scoring dimensions plus 6 cap rules, scored out of 100 and emitted as reward = score / 100.
- Dimensions: verdict correctness, evidence support, analysis quality and clarity
- Only web passages the model actually retrieved and cited count as evidence; with no valid citation the total is capped
- Wrong target, core reasoning errors and unverifiable citations each trigger their own score cap
- The judge uses the fact-checker's full analysis as reference; reference articles and judge code ship with the complete data package
05
Model results
Frontier-model results under the same search tools, budget and judge.
| Model | Mean score / 100 | Mean time per task | Verdict accuracy |
|---|---|---|---|
| DeepSeek V4 Flash20 sampled tasks · three-model test | 76.76 | 47.2 s | 58.8% |
| DeepSeek V4 Pro20 sampled tasks · three-model test | 73.24 | 70.5 s | 64.7% |
| Seed20 sampled tasks · three-model test | 70.88 | 94.2 s | 64.7% |
Measured with the same search tools, budget and judge. All three models get the verdict right less than 65% of the time: searching and writing a rationale is not the same as judging correctly.
06
Specification
| Task ID | FACTCHECK-2350C1D7 |
|---|---|
| Task format | Harbor task.toml · schema 1.4 |
| Package version | 1.0.0 |
| Agent time limit | 15 min |
| Verifier time limit | 5 min |
| Agent container | 1 vCPU · 512 MB memory |
| Tools | search · open · find · submitExecuted by a controlled tool gateway; the agent has no shell or local file access |
| Search budget | 5 searches · 8 reads · 50,000 visible characters · 13 tool rounds |
| Evidence cutoff | 2026-08-24Search results are filtered at the cutoff |
| Instruction length | 274 characters |
| Dataset | FactCheck-200 · 200 tasks |
| Search service | Tavily raw_content, filtered at the cutoff |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support