Evals
Design Engineer · Product, frontend architecture & systems
I led a customer POC from raw evaluation results to a multi-tenant investigation product. It became a core Labelbox service—and produced a reusable system that now helps teams and agents build faster.
- Team
- Design Engineer lead + research engineer
- Scope
- Research · Product · Frontend · Systems
- Outcome
- Customer POC → core service
At a glance
Trajectory
One customer POC
Compare one unreleased model with frontier systems.
Core service
Used by Labelbox's top frontier AI customers.
The product also became the proving ground for a reusable frontend system.
Where it started
The first version proved the idea—and exposed the real problem.
Our early tables and histograms answered “what moved?” Customers still could not find where a regression lived, which failures repeated, or which example explained it. Meanwhile, FDEs repeatedly simplified raw data and delivered each run by hand.
What the POC exposed
Customer view
Shallow detail
Processed summaries showed the score, not the root cause.
FDE workflow
Manual repeat
Every run meant simplifying, checking, and delivering data again.
Frontend path
~1GB ceiling
The detailed JSON customers asked for broke the original load path.
The next version could not be another chart. It had to change the workflow.
The mandate
I had to preserve raw detail without making the product unusable.
The product needed granular results, responsive exploration, secure customer boundaries, and a human checkpoint before publishing. I led the complete application end to end while partnering with a research engineer on processing the raw evaluation data.
My mandate
Lead the product end to end.
Customer workflow, UX, frontend architecture, tenancy, authorization, delivery, and the system that followed.
Detail
Raw task-level results
Performance
Responsive exploration
Boundaries
Multi-tenancy + RBAC
Trust
Human verification
A research engineer partnered with me on raw-data processing.
The turning point
The 1GB JSON forced us to redesign the delivery path.
Raw runs land in GCS. A scheduled or manual scan discovers them; Cloud Run, caching, and large-data processing keep the application responsive. FDEs verify core metrics, then assign runs through RBAC-protected customer projects. Once model results exist, Evals can surface them in roughly an hour—even when the model run itself took days.
How the work evolved
01
Prove
Processed JSON made the comparison visible.
02
Break
~1GB of detail exposed the original load ceiling.
03
Rebuild
Raw discovery, verification, and assignment replaced repetition.
04
Scale
The POC became a core service and a reusable system.
Admin · Ingestion & assignment
AdminUnassigned runs
3
Assigned
3
Raw volume
4.0 GB
Post-run delivery
~1 hour
GCS source of truth
gs://evals-raw
1,284 objects · us-central1
gs://evals-derived
402 objects · us-central1
Cron / manual scan
Nightly at 02:00 · or trigger now to discover new prefixes under evals-raw.
Last automated run · Tonight 02:00 UTC
Discovered runs
Verify core metrics, then assign to a customer project
| Run | GCS path | Discovered | Size | Tasks | Status | Customer | Category | |
|---|---|---|---|---|---|---|---|---|
| terminal-suite-2026-03-12 | gs://evals-raw/northstar/2026-03-12/run_8841/ | Tonight · 02:14 | 842 MB | 120 | New | — | — | |
| voice-probe-batch-77 | gs://evals-raw/harbor/2026-03-12/voice_77/ | Tonight · 02:16 | 318 MB | 64 | New | — | — | |
| tool-horizon-long | gs://evals-raw/harbor/2026-03-12/tools_long/ | Tonight · 02:21 | 980 MB | 180 | Ready | — | — |
2 selected
Designing the investigation
The interface followed the questions researchers asked.
Start broad, isolate the change, group the repeated failures, then inspect the raw example. Gestalt principles shaped the hierarchy; the customer workflow shaped the sequence.
One investigation, not four tools
Compare
Keep identity stable
Regress
Make change dominant
Cluster
Group repeated failures
Inspect
Open the raw example
Comparison had to stay calm while the dimensions multiplied.
Customer conversations showed that the overview still mattered—but only as the start of an investigation. Consistent color and marks preserve identity across views, while hierarchy keeps the initial ranking readable as dimensions change.
Bench · Terminal Suite
Model EvalRelease Date vs Score · Model Evaluation
Plotly · hover points · click legend to show/hide
Leaderboard · 20 configs · 12 series
| # | Model | Config | Accuracy | Cost/Task | Total Cost | TTLT | TTFT | OTPS | Tokens | Tasks |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | ◆F.Alpha | xhigh | 79.8% | $1.42 | $170 | 48.2s | 0.82s | 62 | 184.2k | 120 |
| 2 | ◆F.Alpha | high | 77.1% | $1.18 | $142 | 41.0s | 0.71s | 58 | 162.4k | 120 |
| 3 | ●B.Opus | xhigh | 76.4% | $1.55 | $186 | 52.1s | 0.91s | 49 | 201.1k | 120 |
| 4 | ●B.Opus | high | 74.2% | $1.21 | $145 | 44.6s | 0.78s | 52 | 178.8k | 120 |
| 5 | ◆F.Alpha | mid | 74.0% | $0.92 | $110 | 34.5s | 0.60s | 65 | 138.0k | 120 |
| 6 | ◆F.Beta | high | 73.5% | $1.05 | $126 | 39.2s | 0.69s | 60 | 156.0k | 120 |
| 7 | ●B.Sonnet | high | 72.9% | $0.88 | $106 | 36.4s | 0.64s | 71 | 142.2k | 120 |
| 8 | ▲C.Pro | high | 71.6% | $0.74 | $89 | 31.2s | 0.58s | 66 | 131.0k | 120 |
| 9 | ●B.Opus | mid | 71.0% | $0.95 | $114 | 38.0s | 0.70s | 55 | 149.0k | 120 |
| 10 | ●B.Sonnet | mid | 70.1% | $0.61 | $73 | 28.9s | 0.52s | 78 | 118.4k | 120 |
| 11 | ■NW3 | high | 69.8% | $0.52 | $62 | 27.5s | 0.49s | 74 | 105.6k | 120 |
| 12 | ●B.Haiku | high | 68.9% | $0.33 | $40 | 20.6s | 0.36s | 96 | 79.2k | 120 |
| 13 | ▲C.Pro | mid | 68.5% | $0.49 | $59 | 24.1s | 0.44s | 80 | 98.0k | 120 |
| 14 | ■NW3 | mid | 67.0% | $0.38 | $46 | 22.0s | 0.41s | 85 | 88.4k | 120 |
| 15 | ▲C.Flash | high | 66.2% | $0.28 | $34 | 18.4s | 0.31s | 110 | 72.0k | 120 |
| 16 | ▲C.Pro | low | 64.8% | $0.31 | $37 | 19.2s | 0.35s | 92 | 68.0k | 120 |
| 17 | ■NW3F | high | 64.4% | $0.19 | $23 | 14.8s | 0.28s | 128 | 61.2k | 120 |
| 18 | ⬡Harbor | strict | 63.2% | $0.41 | $49 | 25.4s | 0.47s | 70 | 91.0k | 120 |
| 19 | ◇F.Lite | mid | 61.8% | $0.14 | $17 | 12.1s | 0.22s | 140 | 48.6k | 120 |
| 20 | ⬡Quarry | low | 58.6% | $0.09 | $11 | 9.8s | 0.18s | 155 | 38.4k | 120 |
Regressions mattered more than another absolute score.
Researchers were looking for what broke between runs. Figure–ground moves deltas and outliers forward; filters and dense tables preserve the context needed to decide where to investigate next.
Bench · Terminal Suite
Model EvalMetrics Explorer
Select a metric to rank all configurations · surface regressions
Click to show/hide
Thousands of failures became a path to one example.
Instead of presenting unrelated failed rows, the UI groups recurring modes by proximity. A researcher can choose a cluster, inspect a task, and open the underlying failure, metrics, tokens, and runtime without losing context.
Bench · Terminal Suite
Model EvalOutcome by config
Errors and outcomes · Plotly
| Model | Config | Task ID | Category | Difficulty | Result |
|---|---|---|---|---|---|
| ◆F.Alpha | high | adaptive-sampler-04 | scientific-computing | medium | Schema / format miss |
| ◆F.Beta | high | report-normalize-11 | data-wrangling | easy | Schema / format miss |
| ◆F.Beta | high | long-horizon-07 | tool-use | hard | Resource exceeded |
| ◇F.Lite | high | constraint-solve-02 | reasoning | hard | Reasoning miss |
| ●B.Opus | high | unit-suite-19 | coding | medium | Test failure |
| ●B.Opus | high | adaptive-sampler-20 | scientific-computing | medium | Schema / format miss |
| ●B.Sonnet | high | report-normalize-21 | data-wrangling | easy | Schema / format miss |
| ●B.Sonnet | high | long-horizon-23 | tool-use | hard | Resource exceeded |
| ●B.Haiku | high | constraint-solve-26 | reasoning | hard | Reasoning miss |
| ▲C.Pro | high | unit-suite-29 | coding | medium | Test failure |
| ●B.Haiku | high | adaptive-sampler-04-b | scientific-computing | hard | Schema / format miss |
| ▲C.Pro | high | long-horizon-07-b | tool-use | medium | Resource exceeded |
| ▲C.Flash | high | constraint-solve-02-b | reasoning | hard | Reasoning miss |
| ■NW3 | high | unit-suite-19-b | coding | medium | Test failure |
14 tasks · click a row to inspect the underlying example
Systemizing the work
The patterns were too valuable to leave inside one product.
Teams were building more prototypes, but each one interpreted the brand differently. I turned the interaction and visualization patterns from Evals into reusable, accessible building blocks for prototypes and production products.
Data table
Dense evaluation results with filtering, selection, and detail.
| Series | Dataset | Accuracy | Delta | Status |
|---|---|---|---|---|
| Reasoning | 82.4% | −4.8 | Regression | |
| Tool use | 79.1% | +1.6 | Passed | |
| Coding | 76.8% | −0.7 | Review | |
| Long horizon | 72.5% | +2.1 | Passed | |
| Reasoning | 69.8% | −1.2 | Review |
Selected Frontier Alpha · keyboard focus and row selection included
Agents needed constraints in context—not another document.
Storybook remained the human contract. Skills and rules added an agent-readable layer: they explain which components to use, how to compose them, and where the system’s boundaries are. The same library now supports multiple products, including Labelbox Recursion.
The system evolved
Docs explain
Storybook gives people examples, states, and implementation guidance.
Skills constrain
Rules give agents brand decisions, restrictions, and limitations in context.
The same library now supports multiple prototypes and production products, including Labelbox Recursion.
What changed
We changed both the product and how the company builds.
Evals became a core service used by Labelbox’s top frontier AI customers. Raw ingestion unlocked the granular investigation customers asked for, while the shared library sped up later prototypes and production work—including Recursion—without sacrificing accessibility or brand consistency.
Product outcome
POC → core service
From one frontier AI company to Labelbox's top frontier AI customers, with raw results available in about an hour after a run completes.
Company leverage
One-off → reusable
Components, patterns, skills, and rules now speed later prototypes and production products while preserving the brand.
Stack
Visuals shown are recreated with synthetic data to protect proprietary customer information and internal evaluation datasets.
Broader Labelbox work