For the complete documentation index, see llms.txt. This page is also available as Markdown.

Evals

Evals help you check whether an agent's output meets the requirements you define. You create a set of short acceptance criteria, run it against one or more agent runs, and obtain a pass/fail result for every criterion together with an overall score.

Evals are available for every agent except the default workspace agent. New agents are ready for Evals automatically. Existing agents become ready when you create or run an eval from their Evals tab.

How Evals work

An eval judges a full agent run against the acceptance criteria in an eval set. The result includes:

  • A pass or fail verdict for each criterion

  • A short explanation for each verdict

  • A final score showing the percentage of criteria that passed

Criteria and the run history are saved with the eval result. If you edit an eval set later, earlier results continue to show the criteria that were used when those evaluations ran.

Creating an eval set

  1. Go to Agents in your Toolhouse dashboard.

  2. Open the agent you want to evaluate.

  3. Select the Evals tab between Details and Test.

  4. Click New eval set.

  5. Give the set a title, or keep the default.

  6. Add one acceptance criterion per line item.

  7. Choose when the set should run, then click Save set.

Each eval set has its own enabled switch. You can create multiple sets for the same agent, with different criteria and schedules. Disabling a set stops future evaluations while keeping its previous results available. Deleting a set is a soft delete, so its previous results remain available as historical records.

Writing acceptance criteria

Write criteria as clear, observable requirements. Keep each item focused on one behavior.

For example:

  • The response answers the user's question directly.

  • The response uses the connected account when the task requires it.

  • The response does not invent unavailable information.

In the criteria editor:

  • Press Enter to finish the current item and create the next one.

  • Press Shift + Enter to add a new line inside the current item.

You can ask Toolhouse to suggest criteria from the agent's saved system prompt. Click Generate from prompt in an eval set editor. The suggestions are short requirements describing what the prompt says the agent should do. Review and edit the suggestions before saving them.

Choosing when to run an eval

An eval set can run in either or both of these ways:

  • After every completed output — evaluate each completed run for the agent as soon as the agent completes its task.

  • Cadence — evaluate completed runs from each hourly or daily interval.

If an interval has not run within the hourly or daily interval, Toolhouse will still evaluate the latest completed run for that agent instead. This is useful to observe subtle parameter or tokenizer change from the model providers.

Generating synthetic runs

An agent that hasn't run yet will have nothing to evaluate until its first run completes. In this situations, you can create a synthetic run agent, whose task will be to interact with the agent you want to evaluate.

  1. Ensure the agent you want to test has an Agent identifier. Copy that value.

  2. Open Agents and select Create agent.

  3. In the system prompt, instruct the agent to interact with the agent you want to test. Ensure to provide precise instructions on how the interaction should work. For example, if you need the agent to mimic a frustrated customer, you should instruct the agent to do so.

  4. Also in the system prompt, reference the Agent identifier from step 1.

  5. In the Integrations section, select Add integration, then select Call agent.

  6. Save changes.

Your newly created agent will interact autonomously with the agent you want to eval. If you choose to eval your agent after every completed run, you will see the eval scorecards populating in the Evals log.

Running an eval manually

Use Run eval on an enabled eval set to evaluate the latest completed agent run. You can also choose a number of recent runs, or run the eval for a specific run opened from Logs.

To evaluate a run from Logs:

  1. Open Logs and select a completed run.

  2. Find the Evals row in the run information grid.

  3. Click the small evaluate button.

  4. Choose an enabled eval set and submit the evaluation.

The Evals row shows Not evaluated yet when no completed evaluation exists. When one evaluation exists, its score is clickable and opens the result in a side panel. When a run has multiple evaluations, the row shows the number of evaluations; click it to open the Evals results page filtered to that agent.

Reviewing results

Open Evals from the sidebar or the command palette with Cmd + K. The results page lets you:

  • Filter by agent

  • Filter by an eval set, including all sets for the selected agent

  • Filter by a date range

  • Sort by evaluation date or score

  • Open any result in a side panel

Results load as you scroll, so you can continue browsing without using page navigation controls. The side panel shows the eval status, score, saved criteria, individual verdicts, explanations, and a link back to the agent run.

Suggestions for useful eval sets

You can create separate sets for separate goals. For example:

Eval set
Example criteria

Answer quality

The response is correct and directly answers the request.

Support policy

The response follows the escalation policy and does not promise unsupported actions.

Data workflow

The response uses the required integration and reports the resulting data clearly.

Separate sets make it easier to understand which part of an agent's behavior needs improvement. You can enable a lightweight set after every output and use a more detailed set for hourly or daily review.

Frequently asked questions

Can I evaluate the same agent in different ways? Yes. Create multiple eval sets for the agent and configure each set independently.

Can I change criteria without changing old results? Yes. Existing results preserve their original criteria. New evaluations use the current saved criteria.

Can I evaluate a run more than once? Yes. Run an eval manually again or use another eval set. Each evaluation appears as its own result.

What happens when an agent has no run in a cadence window? Toolhouse uses the latest completed, non-archived run for that agent. If the agent has never completed a run, the window is skipped.

Where can I see the individual criterion decisions? Open a result from the Evals page, or click a single score in the Evals row on a Logs run. The side panel lists every criterion and its verdict.

Last updated