Product Pitch 1-Pager: Standardized Automated Evals Platform

← Back to writings

🤖AI Transparency Statement: This article was fully written by me, then went through basic grammar checks and a light editing feedback loop on clarity and strengthening the positioning with Claude Opus 5.

Epistemic Status: I have fairly high confidence in the overall approach and direction of this based on my domain expertise in online A/B experimentation, where the industry faced similar challenges and standardized towards a common framework. Where I’m less confident currently is in the details of applying it specifically to technical AI safety and evals work. I’m writing this after my own early evaluation of automated evals tooling, two user interviews with members of technical staff, a lot of context gathered from BlueDot AI Safety courses as well as EAGxBerkeley. I would be happy to discuss further with anyone to expand and improve my understanding.

Intro

Technical AI safety and evals run by researchers and members of the technical staff are struggling to keep up with the rate of development of advanced capabilities of models and functional expansions like multi-agent interactions. A centralized automated evals infrastructure and tooling creates an environment to speed up and improve trustworthiness of evals by standardizing the process, and making it easier to use. This would build on top of an existing meshwork of tools created and shared amongst orgs, and also developed internally.

About Me

I spent 3.5 years launching and maturing Expedia Group’s internal data platform for online A/B experimentation, doubling testing volume two years in a row, reducing analysis time by 40%, and preventing millions in losses from underperforming releases. We achieved this through a centralized platform and program work (one example) that democratized usage, particularly by shifting our primary user from analysts and scientists to product managers with less statistical backgrounds. See more here.

Problem Statement

Evals are struggling to keep up with the pace of frontier model development, changes in application like multi-agent applications. Researchers and their teams are stuck doing high-touch work to set up, analyze, and log evals, and then share that information out. Technical teams repeatedly create bespoke solutions and custom libraries on top of a fragmented ecosystem of infrastructure and tools. Both roles are trying to implement ways to address known large issues like scheming, cheating, and sandbagging. The industry is shaken by evals incidents like OpenAI Hugging Face which wasn’t discovered for days and then took METR six days on premises as an independent investigator to run their post-mortem at a high effort cost. Eval setups, analysis, reporting are run differently across different teams and orgs and can produce work that is viewed as more qualitative or lacking rigor. It’s challenging to audit and recall data then understand it for meta-analysis or by other roles like policymakers.

Solution

Centralized evaluations infrastructure and core software platform that provides standard functionality while being highly modular and flexible by supporting the ecosystem of community-driven or internally developed plugins, packages, prompts, measures, standards, and more. This is a standardized system used within and across AI safety organizations to generate consistent results that allow for easy review, analysis, and comparison. Simplified user experiences democratize tooling access, allowing other roles and backgrounds to run and review evaluations without high-touch researcher support.

Details

Infrastructure – Defines consistent compute, data systems, and data structures to make it easy to store everything and easily recall it both into tooling, and any manual querying or analysis. Needs to have clear standards for how modular tools plug in and work. Built to rapidly evolve to meet new demands and support new research. Highly flexible and modular to support contributions from anyone.

Tooling – A core platform provides a user experience that guides the workflows through these stages:

Discovery – Easily find what work has already been completed, and what its results were to inform future work.
Meta-Analysis – Understand trends, find gaps across multiple evals run at different times.
Manage – Organize the chains of work being performed, how evals and experiments can be derived or linked to each other, or driving a goal or objective.
Audit – Within an eval or a sample, humans review the setup, analysis and more to validate high-quality work.
Define – Set up an eval, its goals, measures, and other configuration details.
Run and Collect – Orchestrates the eval and stores all data for retrieval in later phases.
Runtime Monitoring – Detects anomalies, dangerous behaviors, or failures without waiting for the eval to complete.
Analyze – Understand what happened with an eval, or within a sample in order to make a decision.
Log – Capture the decision and any insights gained.
Socialize – Easy for others outside the evals team to review and understand eval setup and results.

Development

Solution and details will be expanded and refined by performing additional user interviews, and identifying lead organizations or lead users to work closely with, potentially through colocation. This will define a strategy and roadmap with quick iterative steps to begin developing the highest-impact items that can be delivered quickly.

Conclusion and Call-to-Action

A standardized and centralized automated evals infrastructure and tooling can democratize access, create more consistent work, and allow rapid adoption of emerging capabilities. This will produce more rigorous and trustworthy results, increase overall throughput of evals, reduce bespoke and repeated technical staff work, while decreasing researcher time spent on evals allowing them to focus on higher-impact work. This leads to reduced infrastructure and engineering costs, evals catching up with the capabilities and pace of development of frontier models, while researchers are freed up to drive more research.

This approach has already been validated in industry with online A/B experimentation.

Who should build this for AI safety? And how? Is this the right shape and size of a solution? Let’s discuss.

Appendix

Theory of Change

Added September 2026

This is an expansion on my original idea that I was inspired to share after discovering Theory of Change Builder on the Effective Altruism Forums. I liked how it supported stepping through and expanding out my original idea while also helping me get specific and scoping out some elements. Adding it to the appendix in the spirit of keeping the 1-pager close to its ideal length.

🤖AI Transparency Statement: Everything below this point was created through Theory of Change Builder and Claude Opus through my inputs, and then also edited by me afterwards.

Explore my full current theory of change for this product as an interactive visual. Note that I can’t control the spacing in the share, which makes the lines heavily overlap. There are some cool interactions if you click onto the tiles and their info icons.

Summary:

This theory of change envisions how a standardized automated evals platform gets organizations running consistent evals at scale. The end of the chain is evaluation evidence trustworthy enough to carry weight in technical AI safety work and in policy frameworks.

Inputs: Funding and a team that covers platform engineering, security, evals domain knowledge, and enough relationship-building to bring the first organizations on.
Outputs: A modular, API-first platform with adapters into the eval frameworks teams already use, contributor tooling that beats rolling your own, a clear licensing and access-control framework, and a schema the community defines.
What has to be true first: Organizations trust it enough to run sensitive evals on it. It fits the workflows they already have. Contributors get something real back for contributing. Results come out structurally comparable.
Then it gets used: Developers, auditors, and regulators run standardized evals on it as a matter of course, researchers publish their evals into the shared library, and cross-organization meta-analysis becomes routine.
A standard emerges: An evaluation quality benchmark that decision-makers reference.
The goal: AI deployment and governance decisions are consistently informed by rigorous, comparable evaluation evidence.
The impact: Severe and large-scale harms from AI systems to humans are avoided.

Where this is weakest:

I scored my confidence in each link. The three I trust least:

Landing a first customer willing to be named publicly (35%): The whole trust argument rests on a reference case, and that assumes cold outreach lands one in a reasonable timeframe and that the results are the kind buyers care about.
A community schema that organizations actually adopt (40%): This assumes key orgs engage with a working group run by someone they have no prior relationship with, and that the output is neutral enough for multiple orgs to buy into.
Decision-makers referencing the standards once they exist (40%): This assumes a credible standards body picks it up and that the reference is actually adopted by users.

There’s an additional one worth naming even though it sits outside the chain I control: whether decision-makers act on better evaluation evidence at all. The platform can make the evidence rigorous and comparable. It cannot make anyone use it.