Quantitative analysis · Fifth Order
Backtest Reliability
Models translate trading instructions into executable backtests, where small implementation choices can materially change the result. Evaluation checks the resulting output against the specified calculation, focusing on reliable rule interpretation and numerical agreement rather than persuasive explanations. Lower error is better. Bars use a nine-decade logarithmic scale so small errors and very large deviations remain visible together without flattening the cohort.
Model results
13 configurationsRankModelScore
- 110.00
Qwen3 Max - 217.78
Nova Premier - 331.62
DeepSeek-V3.2 - 456.23
Grok 4 - 51,000.00
Llama 3.1 Nemotron Ultra - 61,778.28
Claude Sonnet 4.5 - 73,162.28
GPT-5.2 - 856,234.13
Llama 4 Maverick - 9100,000.00Claude Opus 4.5
- 10177,827.94
Gemini 3 Pro Preview - 113,162,277.66
Mistral Large - 125,623,413.25
Command A - 1310,000,000.00GPT-5.1 Codex Max
Numeric table · 13 configurations
| Rank | Model / setting | Score |
|---|---|---|
| 1 | Qwen3 Max | 10.00 |
| 2 | Nova Premier | 17.78 |
| 3 | DeepSeek-V3.2 | 31.62 |
| 4 | Grok 4 | 56.23 |
| 5 | Llama 3.1 Nemotron Ultra | 1,000.00 |
| 6 | Claude Sonnet 4.5 | 1,778.28 |
| 7 | GPT-5.2 | 3,162.28 |
| 8 | Llama 4 Maverick | 56,234.13 |
| 9 | Claude Opus 4.5 | 100,000.00 |
| 10 | Gemini 3 Pro Preview | 177,827.94 |
| 11 | Mistral Large | 3,162,277.66 |
| 12 | Command A | 5,623,413.25 |
| 13 | GPT-5.1 Codex Max | 10,000,000.00 |
Methodology
Error measures deviation from the reference calculation. Lower values rank first. The chart uses a nine-decade logarithmic scale to keep large deviations visible.