Thin material, no room to skip judgement
Two worksheets, 8.5 KB, no cross-checking source. Every conclusion has to come from these two series; anything missing becomes a stated assumption or limitation.
Operating Plan and Earnings of an E-commerce Platform
KTHY-FIN-MEMO-9211 · Operating plan · operating profit path · household position
Two worksheets only: one operating profit and margin path, one commitment and growth series. Join the two to judge how far the operating plan supports earnings, give an explicit position recommendation with its evidence strength, and add quantified sensitivity.
E-commerceProfitabilityPosition adviceSensitivity
11 files · 2 worksheets
2worksheets
17scoring criteria
16agent turns in the longest reference rollout
703Kcumulative prompt tokens in one rollout
01
Highlights
Three deliverables, one story
The workbook, the memo and the JSON must carry the same headline conclusions, and key derived figures must be formulas that reference the source data inside the workbook.
Label your own evidence strength
The instruction asks for a recommendation plus its evidence strength. Separating what you verified from what you infer is itself a scoring point.
Three-model mean score 0.5811
Three models answered under the same criteria for a mean of 0.5811. With only two worksheets, every run finished inside 20 turns: the test is judgement, not retrieval.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 1,515 chars
You are a family financial planner at a family office. The client household is discussing its position in an e-commerce platform listed in Hong Kong; your conclusion decides whether the household buys, adds, or holds, and roughly how much to allocate.
Build an earnings judgement that can support the investment discussion from the material provided, and clearly label source facts, your own calculations and your judgement.
Task
- Assess how far the company's operating plan supports the profitability judgement, naming the main support and the main uncertainty.
- Give an explicit position recommendation and state the strength of the evidence behind it.
- Provide at least one quantified sensitivity analysis, stating which parameters move, which stay fixed, and what later observation would change the recommendation.
NOTEEnglish translation of the original Chinese instruction; the workbook blocks, the five memo sections and the JSON key requirements are omitted.
03
Attachments
Everything visible inside the agent container, with statistics generated from the task package. Original files ship with the full sample package.
| File | Type | Size | Bytes |
|---|---|---|---|
| KTHY_业务底稿.xlsxOperating workpaper: operating profit and margin path (2019A–2028E), commitments and revenue growth including the quarterly series | xlsx | 2 sheets | 8 KB |
04
Scoring design
17independent scoring criteria
16 scoring items1 penalty items
Full criteria text, weights and reference answers ship with the complete data package.
17 criteria: 16 binary plus 1 five-level item, including a structure and organisation dimension.
- 17 criteria: 16 positive scoring items and 1 penalty items (negate)
- Each criterion is judged separately by an agent judge (individual mode), reproducible and auditable
- Judge credentials are injected only into the verifier; the agent container has no scoring channel
- Full criteria text and reference answers ship with the complete data package
Professional standards3
Numerical accuracy3
Argument quality3
Instruction following2
Structure2
Conclusion correctness2
Factual faithfulness2
05
Model results
Trajectory statistics from frontier models in real agent harnesses. One turn is one model call; the context curve shows how state accumulates over a long-horizon task.
| Model | Agent turns | Tool calls | Cumulative prompt tokens | Peak context | Time | Result |
|---|---|---|---|---|---|---|
| Claude Opus 4.8Claude Code · Harbor | 12 | 703K | 74K | — | Completed | |
| GPT-5.6-solCodex · Harbor | 14 | 413K | 35K | — | Completed | |
| Qwen3.8-MaxClaude Code · Harbor | 13 | 555K | 53K | — | Completed |
06
Specification
| Task ID | KTHY-FIN-MEMO-9211 |
|---|---|
| Task format | Harbor task.toml · schema 1.4 |
| Package version | 1.0.0 |
| Agent time limit | 3 h |
| Verifier time limit | 5 h |
| Agent container | 2 vCPU · 8 GB memory · 30 GB storage |
| Network | Only for agent harness installation |
| Tool set | filesystem · shell · python |
| Scoring isolation | Separate verifier containerJudge credentials are injected only into the verifier and are invisible to the agent |
| Instruction length | 1,515 characters |
| Difficulty | A2 (three-model mean 0.5811) |
| Reference solution | oracle reward 1.000 |
| Material size | 2 worksheets · annual and quarterly series |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support