Errors come in three layers
A table whose columns do not add up, a narrative that disagrees with its own table, and a carry-forward break between adjacent years each need a different check — reading end to end finds none of them.
Annual Report Audit Across Provincial Departments
GEN-CN5-001 · 12 annual reports · table reconciliation · three dispositions
Four departments, three years, twelve annual reports, audited one by one against the national format notice: find internal reconciliations that fail, figures that disagree with the narrative, and carry-forward breaks, then file each report as usable, needs clarification, or needs correction.
Open governmentAnnual report auditCross-table checksDispositions
1313 files
68Kcharacters of material
15scoring criteria
41agent turns in the longest reference rollout
2.63Mcumulative prompt tokens in one rollout
01
Highlights
The disposition has to be executable
For a correction, name the year, the table item and the correct figure. A finding that stops at the department level fails.
Twelve reports means twelve conclusions
Every report needs a verdict; leaving any one out loses points. This is about completeness, not workload.
Three-model mean score 0.3968
Three models answered under the same criteria for a mean of 0.3968, the lowest in the set. Every one of the twelve reports needs a verdict; skipping one loses points.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 1,771 chars
You work in the open-government office of a provincial government. Before consolidating the province's annual open-government reports, the office audits twelve reports from four provincial departments across three years, as the national format notice requires the supervising office to review annual reports before consolidation.
Audit all twelve reports against the format notice and the departments' own reports, produce an audit workbook and an audit opinion for the office head.
The opinion must state
- which reports can go straight into the consolidated summary;
- which need clarification from the department;
- which need correction and republication — naming the year, the item and the correct figure.
Conclude only from the source files; where a source file does not disclose something, write 'not disclosed in the source', and do not infer causes or invent data.
NOTEEnglish translation of the original Chinese instruction; the table of thirteen source files and the required deliverable columns are omitted.
03
Attachments
Everything visible inside the agent container, with statistics generated from the task package. Original files ship with the full sample package.
| File | Type | Size | Bytes |
|---|---|---|---|
| 01_西岭省公安厅2023年政府信息公开工作年度报告.md | md | 5,514 chars | 12 KB |
| 02_西岭省公安厅2024年政府信息公开工作年度报告.md | md | 5,613 chars | 13 KB |
| 03_西岭省公安厅2025年政府信息公开工作年度报告.md | md | 5,516 chars | 12 KB |
| 04_西岭省财政厅2023年政府信息公开工作年度报告.md | md | 5,067 chars | 11 KB |
| 05_西岭省财政厅2024年政府信息公开工作报告.md | md | 5,179 chars | 12 KB |
| 06_西岭省财政厅2025年政府信息公开工作报告.md | md | 5,196 chars | 12 KB |
| 07_西岭省交通运输厅2023年政府信息公开工作年度报告.md | md | 5,169 chars | 12 KB |
| 08_西岭省交通运输厅2024年政府信息公开工作年度报告.md | md | 5,234 chars | 12 KB |
| 09_西岭省交通运输厅2025年政府信息公开工作年度报告.md | md | 6,261 chars | 14 KB |
| 10_西岭省农业农村厅2023年政府信息公开工作年度报告.md | md | 5,530 chars | 12 KB |
| 11_西岭省农业农村厅2024年政府信息公开工作年度报告.md | md | 5,014 chars | 11 KB |
| 12_西岭省农业农村厅2025年政府信息公开工作年度报告.md | md | 4,884 chars | 11 KB |
| 13_政府信息公开工作年度报告格式.mdNational annual-report format notice, full text | md | 3,577 chars | 10 KB |
04
Scoring design
15independent scoring criteria
14 scoring items1 penalty items
Full criteria text, weights and reference answers ship with the complete data package.
15 criteria: 14 binary plus 1 five-level item, with five on conclusion correctness.
- 15 criteria: 14 positive scoring items and 1 penalty items (negate)
- Each criterion is judged separately by an agent judge (individual mode), reproducible and auditable
- Judge credentials are injected only into the verifier; the agent container has no scoring channel
- Full criteria text and reference answers ship with the complete data package
Conclusion correctness5
Numerical accuracy4
Argument quality2
Instruction following2
Professional standards1
Factual faithfulness1
05
Model results
Trajectory statistics from frontier models in real agent harnesses. One turn is one model call; the context curve shows how state accumulates over a long-horizon task.
| Model | Agent turns | Tool calls | Cumulative prompt tokens | Peak context | Time | Result |
|---|---|---|---|---|---|---|
| GPT-5.6-solCodex · Harbor | 33 | 2.63M | — | — | Completed | |
| Claude Opus 4.8Claude Code · Harbor | 28 | 888K | 104K | — | Completed | |
| Qwen3.8-MaxClaude Code · Harbor | 23 | — | — | — | Completed |
06
Specification
| Task ID | GEN-CN5-001 |
|---|---|
| Task format | Harbor task.toml · schema 1.4 |
| Package version | 1.0.0 |
| Agent time limit | 20 h |
| Verifier time limit | 5 h |
| Agent container | 2 vCPU · 8 GB memory · 30 GB storage |
| Network | Only for agent harness installation |
| Tool set | filesystem · shell · python |
| Scoring isolation | Separate verifier containerJudge credentials are injected only into the verifier and are invisible to the agent |
| Instruction length | 1,771 characters |
| Difficulty | A3 (three-model mean 0.3968) |
| Reference solution | all gates 84/84, reward 1.000 |
| Material size | 13 Markdown files · 4 departments × 3 years |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support