One PDF, no second source
Every figure comes from the same English audit report. Anything the report does not disclose must be written as 'not disclosed'; no outside data, no estimates.
Mid-Term Review of Supplementary Funding for a Land Tribunal
GEN-AD6-001 · 81-page audit report · case volume reconciliation · funding advice
About CAD 29 million in supplementary funding over four fiscal years is up for review. One 81-page performance audit report is the only source: reconcile intake, disposition and pending cases, apply the ministry's two rules, and document the source of every figure.
Performance reviewAudit reportCase reconciliationFunding advice
11 PDFs · 81 pages
81pages of original PDFs
16scoring criteria
41agent turns in the longest reference rollout
3.40Mcumulative prompt tokens in one rollout
01
Highlights
The basis is itself a judgement
The auditor's verified figures and the agency's own annual-report figures may differ. Which basis you take, and why, belongs in the stated-basis section.
Both rules must be tested separately
Whether cumulative dispositions reach 100% of intake, and whether pending cases at the second year end exceed the starting level — failing either means suspending and requiring a remediation plan.
Three-model mean score 0.5300
Three models answered under the same criteria for a mean of 0.5300. Switching basis inside the long English audit report is the most common failure here.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 1,322 chars
You are an analyst in the financial planning branch of a justice ministry. Over four fiscal years the province added about CAD 29 million to a land tribunal's funding to speed up dispute resolution and clear the backlog. The branch is now doing a mid-term review to decide whether the remaining two years of funding should continue.
Your material is the auditor general's performance audit report on the tribunal for that year. Prepare a case-volume reconciliation and recommend whether funding should continue.
The branch's review convention
Funding continues if, in the first two full fiscal years of the funding period, both conditions hold; if either fails, the branch recommends suspension and requires a remediation plan.
- Cumulative dispositions over two years ÷ cumulative intake over two years ≥ 100%;
- Pending cases at the end of the second fiscal year are no higher than at the start of the funding period.
Use only figures disclosed in the report; where it does not disclose a figure, write 'not disclosed in the report' and do not estimate or import outside data.
NOTEEnglish translation of the original Chinese instruction; the deliverable section headings and precision requirements are omitted.
03
Attachments
Everything visible inside the agent container, with statistics generated from the task package. Original files ship with the full sample package.
| File | Type | Size | Bytes |
|---|---|---|---|
| OAGO_2024_Ontario_Land_Tribunal.pdfAuditor general's annual report, performance audit chapter (English, 81 pages) | 81 pages | 2.0 MB |
04
Scoring design
16independent scoring criteria
15 scoring items1 penalty items
Full criteria text, weights and reference answers ship with the complete data package.
16 criteria: 15 binary plus 1 five-level item, with four each on numerical accuracy and factual faithfulness.
- 16 criteria: 15 positive scoring items and 1 penalty items (negate)
- Each criterion is judged separately by an agent judge (individual mode), reproducible and auditable
- Judge credentials are injected only into the verifier; the agent container has no scoring channel
- Full criteria text and reference answers ship with the complete data package
Numerical accuracy4
Factual faithfulness4
Conclusion correctness3
Professional standards2
Instruction following1
Structure1
Argument quality1
05
Model results
Trajectory statistics from frontier models in real agent harnesses. One turn is one model call; the context curve shows how state accumulates over a long-horizon task.
| Model | Agent turns | Tool calls | Cumulative prompt tokens | Peak context | Time | Result |
|---|---|---|---|---|---|---|
| GPT-5.6-solCodex · Harbor | 36 | 3.40M | — | — | Completed | |
| Claude Opus 4.8Claude Code · Harbor | 17 | 682K | 62K | — | Completed | |
| Qwen3.8-MaxClaude Code · Harbor | 17 | — | — | — | Completed |
06
Specification
| Task ID | GEN-AD6-001 |
|---|---|
| Task format | Harbor task.toml · schema 1.4 |
| Package version | 1.0.0 |
| Agent time limit | 20 h |
| Verifier time limit | 5 h |
| Agent container | 2 vCPU · 8 GB memory · 30 GB storage |
| Network | Only for agent harness installation |
| Tool set | filesystem · shell · python |
| Scoring isolation | Separate verifier containerJudge credentials are injected only into the verifier and are invisible to the agent |
| Instruction length | 1,322 characters |
| Difficulty | A2 (three-model mean 0.5300) |
| Reference solution | all gates 111/111, reward 1.000 |
| Material size | 1 PDF · 81 pages · 2.0 MB |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support