A rumour from Chinese social media
The claim comes from a simplified-Chinese X post, while the evidence is mostly in Spanish- and English-language media; the model chooses its own search languages and keywords.
The Claim That Spain Will Replace the Dollar with the Renminbi
FACTCHECK-25786D41 · Open-web fact verification · cross-language financial rumour
During US–Spain trade tensions, a Chinese-language social post claimed Spain had announced it would replace the dollar with the renminbi. The model must search across languages for official statements and authoritative reporting to judge the rumour.
Fact-checkingFinancial rumourCross-languageEnglish
5 / 8searches / page reads
50Kvisible characters at most
4scoring dimensions + 6 cap rules
3frontier models tested
< 65%frontier-model verdict accuracy
01
Highlights
Scope the claim first
Who announced it, what was announced, and whether it became policy — the check requires breaking the claim down before finding evidence for each part.
Citations must be verifiable
Every piece of evidence must match a passage the model actually read; fabricated or unmatched citations directly lower the score cap.
Honest abstention is allowed
When evidence is insufficient the model may abstain. Abstaining earns analysis and clarity credit but not full evidence credit, so the reward encourages neither guessing nor blanket abstention.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 179 chars
Verify the public claim with the controlled search/read tools and submit a verdict with evidence.
Claim
Spain has announced it will use the renminbi to replace the dollar.
Context
A simplified Chinese X post published on July 16, 2026, amid US-Spain trade tensions after Trump's NATO remarks.
Evidence cutoff
2026-08-06. Search results are filtered to this date.
NOTEThis is the model's entire public input; tool definitions and the submission format are omitted.
03
Search environment
5searches (raw page text, no generated summaries)
8page reads
50Kvisible characters at most
2026.08.06evidence cutoff; later pages are filtered out
No attachments: all evidence comes from the model's own open-web searching and reading. The full tool-call record (queries, pages opened, passages cited) is saved with every attempt for traceability.
04
Scoring design
4scoring dimensions
+ 6 cap rules
Full criteria text, weights and reference answers ship with the complete data package.
4 scoring dimensions plus 6 cap rules, scored out of 100 and emitted as reward = score / 100.
- Dimensions: verdict correctness, evidence support, analysis quality and clarity
- Only web passages the model actually retrieved and cited count as evidence; with no valid citation the total is capped
- Wrong target, core reasoning errors and unverifiable citations each trigger their own score cap
- The judge uses the fact-checker's full analysis as reference; reference articles and judge code ship with the complete data package
05
Model results
Frontier-model results under the same search tools, budget and judge.
| Model | Mean score / 100 | Mean time per task | Verdict accuracy |
|---|---|---|---|
| DeepSeek V4 Flash20 sampled tasks · three-model test | 76.76 | 47.2 s | 58.8% |
| DeepSeek V4 Pro20 sampled tasks · three-model test | 73.24 | 70.5 s | 64.7% |
| Seed20 sampled tasks · three-model test | 70.88 | 94.2 s | 64.7% |
Measured with the same search tools, budget and judge. All three models get the verdict right less than 65% of the time: searching and writing a rationale is not the same as judging correctly.
06
Specification
| Task ID | FACTCHECK-25786D41 |
|---|---|
| Task format | Harbor task.toml · schema 1.4 |
| Package version | 1.0.0 |
| Agent time limit | 15 min |
| Verifier time limit | 5 min |
| Agent container | 1 vCPU · 512 MB memory |
| Tools | search · open · find · submitExecuted by a controlled tool gateway; the agent has no shell or local file access |
| Search budget | 5 searches · 8 reads · 50,000 visible characters · 13 tool rounds |
| Evidence cutoff | 2026-08-06Search results are filtered at the cutoff |
| Instruction length | 179 characters |
| Dataset | FactCheck-200 · 200 tasks |
| Search service | Tavily raw_content, filtered at the cutoff |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support