How reliably do AI coding agents solve quantitative-finance problems?
Verifiable tasks from quant work, frozen LSEG market data, and error bars on every number. Every configuration of harness, model and reasoning effort attempts each task three times.
By Léon Faro, statistics and quantitative finance.
Suite core10-v0: 10 tasks × 3 runs per configuration, run with Harbor [8]. Ranked by pass^3, the share of tasks solved in all three runs. Brackets are 95% intervals from a cluster bootstrap over tasks. Rank = 1 + the number of configurations whose pass^3 interval lies entirely above. * marks the cost–reliability frontier (Figure 1).
SIMULATED Numbers from a two-parameter item-response model, seed 20261017. They preview the layout of release 1 and are not results. H is a harness, M a model.
| Rank | Harness | Model | Effort | pass^3 | 95% CI | pass@1 | 95% CI | pass@3 | $/task | $/solved | Time | Steps | Tokens | Cache |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No runs yet. Release 1 follows once all ten tasks pass their oracle runs. | ||||||||||||||
Planned: Claude Code, Codex CLI, OpenCode and Pi as harnesses; frontier and open-weight models; reasoning effort low to xhigh. Cost is the API list price on the run date.
Each task is a container with an instruction, a frozen data snapshot, my reference solution and a verifier that decides pass or fail. A task goes live once its reference solution passes its own verifier on 100 reseeded runs. Solutions, and tasks built on LSEG data, stay private.
Lo–MacKinlay variance ratios VR(q) and robust z*(q) for q = 5, 15, 60 on 1-minute returns, with weekends and gaps handled [6].
Data: LSEG EUR/USD 1-min
Heston paths with full-truncation Euler. The discounted price must stay a martingale, E[exp(−rT) S_T] = S_0, within the Monte Carlo band on unseen seeds.
Data: synthetic
Arithmetic Asian call with the geometric Asian as control variate. Correct price and an honest standard error (coverage check) within a compute budget.
Data: synthetic
Bayesian stochastic-volatility model. Posterior means of φ and σ_η match a long NUTS reference, with R-hat < 1.01 and ESS > 400 reported.
Data: LSEG FX daily
Find and fix a subtle Metropolis–Hastings bug, such as a missing Hastings ratio or Jacobian. The fixed sampler must match an analytic posterior (KS test).
Data: synthetic
GARCH(1,1) with Student-t innovations, one-day 99% VaR. Exceedance count plus Kupiec and Christoffersen p-values.
Data: LSEG, to be decided
Settlement-aware tenor dates (spot lag, holiday calendars, modified following; ON, TN, 1W, 1M) and ACT/365F carry from forward points.
Data: LSEG FX forwards
Find a look-ahead bias hidden in a backtest, such as a centered window or trading on the close that produced the signal. Report the corrected Sharpe ratio.
Data: LSEG, to be decided
Deflated Sharpe ratio and probability of backtest overfitting across 200 strategy trials [7].
Data: synthetic
Calibrate Heston through its characteristic function (Lewis or Carr–Madan) to a volatility surface. Pricing RMSE and parameter recovery.
Data: synthetic surface
With T tasks, n = 3 runs per task and c_t passing runs on task t:
pass@k = (1/T) Σ_t [ 1 − C(n − c_t, k) / C(n, k) ] (1) pass^k = (1/T) Σ_t C(c_t, k) / C(n, k) (2)
(1) is the unbiased estimator of Chen et al. [3]. (2) is the pass^k of τ-bench [4], the chance that k independent attempts all succeed. For n = k = 3 it is the share of tasks solved three times out of three.
Monte Carlo and MCMC answers vary between runs, so for these tasks the verifier is itself a statistical test. An estimate passes when
|θ_hat − θ_ref| ≤ z_(1−α/2) · √(SE_agent² + SE_ref²)
and SE_agent ≤ SE_max (3)
with α fixed per task. The reference solution must pass at least 99 of 100 reseeded runs. If it passes fewer, the tolerance is too strict and the task goes back to build.
pass ~ config + (1 | task).Agents never receive credentials or live data access. Each container holds a frozen data snapshot identified by its hash. Every run writes one record; the values below are examples.
{
"run": "r1/A1/H1-M1-high/2",
"harness": { "name": "…", "version": "…" },
"model": { "id": "…", "effort": "high" },
"data": { "snapshot": "sha256:4be1…09c2",
"window": "2026-10" },
"seed": 20261017,
"verdict": { "pass": true, "stat": "z*(15)", "value": 2.214,
"ref": 2.209, "tol": 0.031 },
"usage": { "usd": 0.31, "seconds": 372, "steps": 41,
"tokens_in": 171204, "tokens_out": 11842,
"cache_hit": 0.87 }
}
Hypotheses, design and analysis plan are committed to the repository before the first run. The commit hash appears here once an experiment is registered.
Same model in Claude Code, Codex CLI, OpenCode and Pi.
One model, effort low to xhigh, cost against pass^3.
With and without a file on day counts, calendars and quoting.
With and without LSEG API documentation in the container.
Textbook parameters against fresh data; Green Book originals against perturbed variants.
Open-weight models (Qwen, DeepSeek, GLM) against frontier models.
numpy and scipy only, against QuantLib installed.
Prompt with and without a verification instruction.
@misc{faro2026sigmabench,
author = {Faro, L{\'e}on},
title = {{$\sigma$-Bench}: How Reliably Do {AI} Coding
Agents Solve Quantitative-Finance Problems?},
year = {2026},
note = {Living report, release 0},
url = {https://sigmabench.dev}
}