OpenAI Introduces the Evals API: Streamlined Model Evaluation for Developers
In a significant move to empower developers and teams working with large language models (LLMs), OpenAI has introduced the Evals API, a new toolset that brings programmatic evaluation capabilities to the forefront. While evaluations were previously accessible via the OpenAI dashboard, the new API allows developers to define tests, automate evaluation runs, and iterate on prompts directly from their workflows.
Why the Evals API Matters
Evaluating LLM performance has often been a manual, time-consuming process, especially for teams scaling applications across diverse domains. With the Evals API, OpenAI provides a systematic approach to:
Assess model performance on custom test cases
Measure improvements across prompt iterations
Automate quality assurance in development pipelines
Now, every developer can treat evaluation as a first-class citizen in the development cycle—similar to how unit tests are treated in traditional software engineering.
Core Features of the Evals API
Custom Eval Definitions: Developers can write their own evaluation logic by extending base classes.
Test Data Integration: Seamlessly integrate evaluation datasets to test specific scenarios.
Parameter Configuration: Configure model, temperature, max tokens, and other generation parameters.
Automated Runs: Trigger evaluations via code, and retrieve results programmatically.
The Evals API supports a YAML-based configuration structure, allowing for both flexibility and reusability.
Getting Started with the Evals API
To use the Evals API, you first install the OpenAI Python package:
pip install openai
Then, you can run an evaluation using a built-in eval, such as factuality_qna
oai evals registry:evaluation:factuality_qna \
--completion_fns gpt-4 \
--record_path eval_results.jsonl
Or define a custom eval in Python:
import openai.evals
class MyRegressionEval(openai.evals.Eval):
def run(self):
for example in self.get_examples():
result = self.completion_fn(example['input'])
score = self.compute_score(result, example['ideal'])
yield self.make_result(result=result, score=score)
This example shows how you can define a custom evaluation logic—in this case, measuring regression accuracy.
Use Case: Regression Evaluation
OpenAI’s cookbook example walks through building a regression evaluator using the API. Here’s a simplified version:
from sklearn.metrics import mean_squared_error
class RegressionEval(openai.evals.Eval):
def run(self):
predictions, labels = [], []
for example in self.get_examples():
response = self.completion_fn(example['input'])
predictions.append(float(response.strip()))
labels.append(example['ideal'])
mse = mean_squared_error(labels, predictions)
yield self.make_result(result={"mse": mse}, score=-mse)
This allows developers to benchmark numerical predictions from models and track changes over time.
Seamless Workflow Integration
Whether you’re building a chatbot, summarization engine, or classification system, evaluations can now be triggered as part of your CI/CD pipeline. This ensures that every prompt or model update maintains or improves performance before going live.
openai.evals.run(
eval_name="my_eval",
completion_fn="gpt-4",
eval_config={"path": "eval_config.yaml"}
)
Conclusion
The launch of the Evals API marks a shift toward robust, automated evaluation standards in LLM development. By offering the ability to configure, run, and analyze evaluations programmatically, OpenAI is enabling teams to build with confidence and continuously improve the quality of their AI applications.
To explore further, check out the official OpenAI Evals documentation and the cookbook examples.
Asif RazzaqWebsite | + postsBioAsif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.Asif Razzaqhttps://www.marktechpost.com/author/6flvq/Salesforce AI Released APIGen-MT and xLAM-2-fc-r Model Series: Advancing Multi-Turn Agent Training with Verified Data Pipelines and Scalable LLM ArchitecturesAsif Razzaqhttps://www.marktechpost.com/author/6flvq/A Code Implementation to Use Ollama through Google Colab and Building a Local RAG Pipeline on Using DeepSeek-R1 1.5B through Ollama, LangChain, FAISS, and ChromaDB for Q&AAsif Razzaqhttps://www.marktechpost.com/author/6flvq/A Step-by-Step Coding Guide to Building a Gemini-Powered AI Startup Pitch Generator Using LiteLLM Framework, Gradio, and FPDF in Google Colab with PDF Export SupportAsif Razzaqhttps://www.marktechpost.com/author/6flvq/Transformer Meets Diffusion: How the Transfusion Architecture Empowers GPT-4o’s Creativity
English
Arabic
French
Spanish
Portuguese
Deutsch
Turkish
Dutch
Italiano
Russian
Romaian
Portuguese (Brazil)