Real data, production constraints
Built on a real China A-share market snapshot: 15,613 initial factor rows and 423 mixed v1/v2/v3 events across 8 batches, with 60-trading-day rolling factors at full Decimal precision.
Transactional Recovery for a Partitioned Factor Store
FACTOR-STORE-RECOVERY-0001 · WAL crash recovery · minimal incremental recompute · exactly-once
Repair a partitioned factor store driven by real market data: handle vendor revisions, deletes and cross-partition moves, recover to a consistent state after a crash injected at any stage, and rewrite only the partitions that changed.
Data engineeringTransaction protocolPythonHidden test set
1313 files · 15,900 data rows
1.5Mcharacters of material
11programmatic checks
—agent turns in the longest reference rollout
0.14frontier-model blind-run reward
01
Highlights
Fault injection, not unit tests
The grader injects a crash at a chosen stage of an 8-stage transaction state machine and checks the WAL, staging and checkpoint boundaries — uncommitted, partially committed, and committed but unacknowledged.
Hard-coding earns nothing
The hidden set uses different dates, securities and crash points. Empty programs, the starter code and submissions that hard-code public results all score 0 in cold-start acceptance; 12 / 12 acceptance checks pass.
Frontier model blind run: 0.14
GPT-5.6-sol scored 0.1405 in an independent blind run: durability was right, but MOVE/tombstone, rolling recompute and both recovery groups all failed. The spread comes from genuine engineering difficulty.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 4,053 chars
/app/repo is a factor store partitioned by trade_date/hash_bucket. It receives vendor revisions, deletes, natural-key moves and schema upgrades, and maintains 60-trading-day rolling factors. The current implementation corrupts state on crashes, batch replays and cross-partition moves, and fully recomputes every batch. Fix the project.
Event semantics
- v1 uses factor/value_date/score, migrated to factor_name/trade_date/value; v2 matches v3 but without new_symbol/new_trade_date.
- Events are ranked by (revision, received_at, source_priority, event_id); a winning DELETE is a durable tombstone, and older revisions must not resurrect the data.
- A MOVE takes effect only if its sort key beats the current winner or tombstone of both the old and the new natural key; otherwise neither side may change.
- event_id is globally exactly-once; duplicates within and across batches must be recorded but must not apply again.
Transaction protocol
Each batch follows the fixed state machine READ → VALIDATE → PLAN → STAGE_RAW → STAGE_DERIVED → STAGE_MANIFEST → COMMIT → CHECKPOINT. The grader injects one fault. On startup the store must use the WAL to distinguish “not started”, “partially committed” and “committed but checkpoint not confirmed”, and roll back or forward. It must not simply delete staging and replay a MOVE that may have been partially committed.
Minimal updates
Only raw/derived partitions whose content changed may be replaced; unaffected partitions must keep the same SHA256, inode and mtime. Rewriting identical content to get around this is not allowed. A clean run, recovery from any fault and a repeated run over the same event stream must produce the same logical state.
Time limit
The agent has 1 hour in total, including reading the task, changing code, local testing and final checks. Python standard library only; no network.
NOTEEnglish translation of the original Chinese instruction; schema definitions for state files and outputs are omitted.
03
Attachments
Everything visible inside the agent container, with statistics generated from the task package. Original files ship with the full sample package.
| File | Type | Size | Bytes |
|---|---|---|---|
| batch_order.jsonBatch commit order | json | 112 chars | 112 B |
| batches/ (8 files)Mixed v1 / v2 / v3 event batches | jsonl | 423 rows | 104 KB |
| calendar.csvTrading calendar | csv | 204 rows | 3 KB |
| initial_raw.csvInitial raw factors | csv | 15,613 rows | 1.3 MB |
| source_policy.csvVendor priority | csv | 3 rows | 50 B |
| symbol_exchange.csvSecurity → exchange mapping | csv | 80 rows | 1 KB |
Plus a starter project in /app/repo (5 files, about 99 lines of Python) to repair.
04
Scoring design
11programmatic checks
+ 3 capability gates / score caps
Full criteria text, weights and reference answers ship with the complete data package.
11 programmatic checks plus 3 capability gates, all computed by a deterministic grader with no LLM judge.
- Checks cover schema and Decimal handling, conflicts and tombstones, exactly-once, atomic MOVE, rolling factors, minimal recompute, both crash recoveries, idempotence and audit
- Hidden core gate, recovery caps and mastery caps: missing any core capability caps the total
- The reference solution scores 1.0 on two random seeds; empty programs, the starter and hard-coded public results score 0
- Grader, hidden inputs and reference solution ship with the complete data package
05
Model results
Trajectory statistics from frontier models in real agent harnesses. One turn is one model call; the context curve shows how state accumulates over a long-horizon task.
0.1405GPT-5.6-sol independent blind-run reward (effort medium)
1.000reference solution reward (two random seeds)
0empty program / hard-coded public results
12 / 12cold-start acceptance checks passed
The blind run was a single agent answering in sequence; its candidate was frozen before the private grader ran. Durability was near perfect, while MOVE/tombstone, rolling recompute, semantic ingest and both recovery groups all scored 0.
06
Specification
| Task ID | FACTOR-STORE-RECOVERY-0001 |
|---|---|
| Task format | Harbor task.toml · schema 1.4 |
| Package version | 1.4.0 |
| Agent time limit | 1 h |
| Verifier time limit | 15 min |
| Agent container | 2 vCPU · 2 GB memory |
| Instruction length | 4,053 characters |
| Difficulty tier | hard (blind run 0.1405) |
| Reference solution | reward 1.000 on both seeds |
| Data source | Real China A-share market snapshot + synthetic transaction event layer |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support