The Jev benchmark, in plain numbers
Every Jev benchmark you will see today is about speed, cost and calibration — not a public accuracy leaderboard. This page explains what the published figures actually mean and how to test them.
What the numbers are
The headline figures are performance, not accuracy. Jev returns a decision in 70 to 500 ms, roughly 40 to 200 times faster than a frontier LLM. On cost, you pay only per input token — output free, a fraction of a cent per decision (see pricing).
- Latency: 70 to 500 ms per call.
- Speed: about 40 to 200 times faster than a frontier LLM.
- Cost: pay per input token, output free — a fraction of a cent per decision (see pricing).
- Throughput: about 250,000 tokens per second and 1,200 requests per minute.
- Context: up to about 64k tokens, with 32k for state plus one longest question.
Calibration is the real Jev benchmark
Because Jev is trained with RLCD for calibration rather than fluency, the meaningful measure is whether its probabilities match reality. Every answer carries per-option probabilities and a confidence, where confidence describes how concentrated the answer is, not whether it is right.
How Jev compares — the models people ask about
TypeSafe hasn't published its own accuracy leaderboard, but the third-party JevBench ranks Jev-class systems on a frozen set of hard typed decisions — and Jev leads it. Here's where the decision models people actually search for land:
| Model | Type | JevBench standing | Best for |
|---|---|---|---|
| Jev 1.13.0 | Hosted, closed (TypeSafe) | #1 | Calibrated, works zero-shot, no infra |
| djev | Open — diffusion, image input | Top open tier | Fast open decisions, native image/camera |
| SemIf / OpenJev | Open — logit-reading / self-host server | Strong open | Self-hosted, Jev-compatible endpoint |
| Laya | Open weights — Apache-2.0, runs on a Mac | Low zero-shot | Ownership & offline, once you fine-tune it |
The headline gap is zero-shot accuracy: on JevBench's hard tier, Jev scores around 74% versus about 34% for Laya out of the box. Laya is the most-downloaded open model, but it's built to be fine-tuned, not used raw — which is why it ranks low un-tuned. Exact scores and ranks move between benchmark versions (the scale itself changed from v1.3 to v1.4), so treat the standings as directional and check the live board for current numbers.
- Benchmark Heaven — live JevBench leaderboard — all 50+ systems, updated continuously
- JevBench (GitHub) — benchmark harness, tasks & scoring rules (MIT)
Run your own Jev benchmark
The only Jev benchmark that matters for you is on your own labelled data. Send a batch of real cases, compare Jev's decisions to your ground truth, and calibrate your confidence thresholds. That turns generic figures into numbers you can trust in production.
- Collect a labelled sample from your own domain.
- Call Jev with your real states and questions.
- Compare decisions to ground truth and tune your thresholds.
See also: Pricing · How to use the API · Playground
Related: Jev vs an LLM · System One model · Jev vs Laya
Run your own numbers
Try your real cases in the playground, then get a key to benchmark Jev at scale on your own data.