Unclosed div`
- Non-HTML content with HTML inside: `Here is some HTML:
test
`
### Contains-Sql
This assertion ensure that the output is either valid SQL, or contains a code block with valid SQL.
```yaml
assert:
- type: contains-sql
```
See [`is-sql`](#is-sql) for advanced usage, including specific database types and allowlists for tables and columns.
### Cost
The `cost` assertion checks if the cost of the LLM call is below a specified threshold.
This requires LLM providers to return cost information. Currently this is only supported by OpenAI GPT models and custom providers.
Example:
```yaml
providers:
- openai:gpt-5-mini
- openai:gpt-5
assert:
# Pass if the LLM call costs less than $0.001
- type: cost
threshold: 0.001
```
### Equality
The `equals` assertion checks if the LLM output is equal to the expected value.
Example:
```yaml
assert:
- type: equals
value: 'The expected output'
```
You can also check whether it matches the expected JSON format.
```yaml
assert:
- type: equals
value: { 'key': 'value' }
```
If your expected JSON is large, import it from a file:
```yaml
assert:
- type: equals
value: 'file://path/to/expected.json'
```
### Is-JSON
The `is-json` assertion checks if the LLM output is a valid JSON string.
Example:
```yaml
assert:
- type: is-json
```
You may optionally set a `value` as a JSON schema. If set, the output will be validated against this schema:
```yaml
assert:
- type: is-json
value:
required:
- latitude
- longitude
type: object
properties:
latitude:
minimum: -90
type: number
maximum: 90
longitude:
minimum: -180
type: number
maximum: 180
```
JSON is valid YAML, so you can also just copy in any JSON schema directly:
```yaml
assert:
- type: is-json
value:
{
'required': ['latitude', 'longitude'],
'type': 'object',
'properties':
{
'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
},
}
```
If your JSON schema is large, import it from a file:
```yaml
assert:
- type: is-json
value: file://./path/to/schema.json
```
### Is-XML
The `is-xml` assertion checks if the entire LLM output is a valid XML string. It can also verify the presence of specific elements within the XML structure.
Example:
```yaml
assert:
- type: is-xml
```
This basic usage checks if the output is valid XML.
You can also specify required elements:
```yaml
assert:
- type: is-xml
value:
requiredElements:
- root.child
- root.sibling
```
This checks if the XML is valid and contains the specified elements. The elements are specified as dot-separated paths, allowing for nested element checking.
#### How it works
1. The assertion first attempts to parse the entire output as XML using a parser (fast-xml-parser).
2. If parsing succeeds, it's considered valid XML.
3. If `value` is specified:
- It checks for a requiredElements key with an array of required elements.
- Each element path (e.g., "root.child") is split by dots.
- It traverses the parsed XML object following these paths.
- If any required element is not found, the assertion fails.
#### Examples
Basic XML validation:
```yaml
assert:
- type: is-xml
```
Passes for: `
Content`
Fails for: `
ContentT-shirtRed`
Fails for: `T-shirt` (missing color element)
Checking nested elements:
```yaml
assert:
- type: is-xml
value:
requiredElements:
- root.parent.child.grandchild
```
Passes for: `Content`
Fails for: `` (missing grandchild element)
#### Inverse assertion
You can use the `not-is-xml` assertion to check if the output is not valid XML:
```yaml
assert:
- type: not-is-xml
```
This will pass for non-XML content and fail for valid XML content.
Note: The `is-xml` assertion requires the entire output to be valid XML. For checking XML content within a larger text, use the `contains-xml` assertion.
### Contains-XML
The `contains-xml` assertion checks if the LLM output contains valid XML content, even if it's not the entire output. It uses the same `value.requiredElements` format as `is-xml`; required element paths are checked from the extracted XML fragment's root.
```xml
Sure, here is your xml:
Content
let me know if you have any other questions!
```
For example, `contains-xml` with `root.child` passes for `Text Content`. If the XML fragment is wrapped, use the wrapper in the required path, such as `wrapper.root.child` for `Content`.
### Is-SQL
The `is-sql` assertion checks if the LLM output is a valid SQL statement.
Example:
```yaml
assert:
- type: is-sql
```
To use this assertion, you need to install the `node-sql-parser` package. You can install it using npm:
```bash
npm install node-sql-parser
```
You can optionally set a `databaseType` in the `value` to determine the specific database syntax that your LLM output will be validated against. The default database syntax is MySQL. For a complete and up-to-date list of supported database syntaxes, please refer to the [node-sql-parser documentation](https://github.com/taozhi8833998/node-sql-parser?tab=readme-ov-file#supported-database-sql-syntax).
The supported database syntax list:
- Athena
- BigQuery
- DB2
- FlinkSQL
- Hive
- MariaDB
- MySQL
- Noql
- PostgresQL
- Redshift
- Snowflake(alpha)
- Sqlite
- TransactSQL
Example:
```yaml
assert:
- type: is-sql
value:
databaseType: 'MySQL'
```
You can also optionally set a `allowedTables`/`allowedColumns` in the `value` to determine the SQL authority list that your LLM output will be validated against.
The format of allowedTables:
```
{type}::{dbName}::{tableName} // type could be select, update, delete or insert
```
The format of allowedColumns:
```
{type}::{tableName}::{columnName} // type could be select, update, delete or insert
```
For `SELECT *`, `DELETE`, and `INSERT INTO tableName VALUES()` without specified columns, the `.*` column authority regex is required.
Example:
```yaml
assert:
- type: is-sql
value:
databaseType: 'MySQL'
allowedTables:
- '(select|update|insert|delete)::null::departments'
allowedColumns:
- 'select::null::name'
- 'update::null::id'
```
### is-valid-function-call
This ensures that any JSON LLM output adheres to the schema specified in the `functions` configuration of the provider. This is implemented for a subset of providers. Learn more about the [Google Vertex provider](/docs/providers/vertex/#function-calling-and-tools), [Google AIStudio provider](/docs/providers/google/#tool-calling), [Google Live provider](/docs/providers/google#function-calling-example) and [OpenAI provider](/docs/providers/openai/#tool-calling), which this is implemented for.
### is-valid-openai-function-call
Legacy - please use is-valid-function-call instead. This ensures that any JSON LLM output adheres to the schema specified in the `functions` configuration of the provider. Learn more about the [OpenAI provider](/docs/providers/openai/#tool-calling).
### is-valid-openai-tools-call
This ensures that any JSON LLM output adheres to the schema specified in the `tools` configuration of the provider. Learn more about the [OpenAI provider](/docs/providers/openai/#tool-calling).
**MCP Support**: This assertion also validates MCP (Model Context Protocol) tool calls when using OpenAI's Responses API. It will:
- Pass if MCP tool calls succeed (output contains "MCP Tool Result")
- Fail if MCP tool calls fail (output contains "MCP Tool Error")
- Continue to validate traditional function tools as before
Example with MCP tools:
```yaml
providers:
- id: openai:responses:gpt-5
config:
tools:
- type: mcp
server_label: deepwiki
server_url: https://mcp.deepwiki.com/mcp
require_approval: never
tests:
- vars:
query: 'What is MCP?'
assert:
- type: is-valid-openai-tools-call # Validates MCP tool success
- type: contains
value: 'MCP Tool Result' # Alternative way to check for MCP success
```
### tool-call-f1
The `tool-call-f1` assertion computes the [F1 score](https://en.wikipedia.org/wiki/F-score) comparing the set of tools called by the LLM against an expected set of tools. This metric is useful for evaluating agentic LLM applications where you want to measure how accurately the model selects the right tools.
This assertion supports multiple provider formats including OpenAI, Anthropic, and Google/Vertex.
The F1 score is the harmonic mean of precision and recall, originally introduced by [van Rijsbergen (1979)](http://www.dcs.gla.ac.uk/Keith/Preface.html) for information retrieval evaluation:
- **Precision** = |actual ∩ expected| / |actual| — "Of the tools called, how many were correct?"
- **Recall** = |actual ∩ expected| / |expected| — "Of the expected tools, how many were called?"
- **F1** = 2 × (precision × recall) / (precision + recall)
This uses **unordered set comparison** — only the presence of tool names matters, not the order or frequency of calls.
Example:
```yaml
providers:
- id: openai:gpt-4.1
config:
tools:
- type: function
function:
name: get_weather
parameters:
type: object
properties:
city:
type: string
- type: function
function:
name: book_flight
parameters:
type: object
properties:
destination:
type: string
tests:
- vars:
query: "What's the weather in NYC and book me a flight to LA?"
assert:
# Require exact match (F1 = 1.0)
- type: tool-call-f1
value:
- get_weather
- book_flight
# Allow partial matches with custom threshold
- type: tool-call-f1
value: ['get_weather', 'book_flight']
threshold: 0.8
```
The `value` can be specified as:
- An array of tool names: `['get_weather', 'book_flight']`
- A comma-separated string: `'get_weather, book_flight'`
The `threshold` defaults to `1.0` (exact match required). Lower thresholds allow partial matches, which is useful during development or when some flexibility is acceptable.
**Scoring examples:**
| Expected Tools | Actual Tools Called | Precision | Recall | F1 |
| ---------------------------- | ------------------------------------ | --------- | ------ | ----- |
| `[get_weather, book_flight]` | `[get_weather, book_flight]` | 1.0 | 1.0 | 1.0 |
| `[get_weather, book_flight]` | `[get_weather]` | 1.0 | 0.5 | 0.667 |
| `[get_weather, book_flight]` | `[get_weather, book_flight, search]` | 0.667 | 1.0 | 0.8 |
| `[get_weather]` | `[book_flight]` | 0.0 | 0.0 | 0.0 |
### skill-used {#skill-used}
The `skill-used` assertion checks normalized provider skill metadata rather than the model's final output. It works well for agent evals where the important question is "did the agent route through the right skill?".
Promptfoo currently populates `metadata.skillCalls` for:
- Claude Agent SDK, by normalizing `Skill` tool calls.
- OpenAI Codex SDK, by inferring skill usage from command text that directly references a local `SKILL.md` path.
Example:
```yaml
assert:
- type: skill-used
value: code-review
- type: skill-used
value:
pattern: 'project-*:*'
min: 1
- type: not-skill-used
value: forbidden-skill
```
Use `skill-used` when provider-level routing evidence is enough. If you also need to verify what the skill actually did, combine it with trace assertions such as [`trajectory:tool-used`](#trajectorytool-used), [`trajectory:tool-args-match`](#trajectorytool-args-match), or [`trajectory:step-count`](#trajectorystep-count).
### trajectory:tool-used {#trajectorytool-used}
The `trajectory:tool-used` assertion checks traced tool steps rather than the model's final output. It works well for agent evals where the important question is "did the agent actually use the right tool?".
:::note
Trajectory assertions require trace data. Enable tracing for the eval and use a provider that emits tool-oriented spans or attributes.
:::
Promptfoo identifies tool names from attributes such as `tool.name`, `function.name`, and Vercel AI SDK telemetry's `ai.toolCall.name`.
Example:
```yaml
tests:
- assert:
- type: trajectory:tool-used
value: search_orders
- type: trajectory:tool-used
value:
pattern: 'search*'
min: 2
max: 3
```
`value` may be:
- A string, such as `search_orders`
- An array of strings, such as `['search_orders', 'compose_reply']`
- An object with `pattern`, `min`, and optional `max`
### trajectory:tool-args-match {#trajectorytool-args-match}
The `trajectory:tool-args-match` assertion checks traced tool-call arguments. Use it when the agent must not only invoke the right tool, but also pass the right parameters.
Example:
```yaml
tests:
- vars:
order_id: '123'
assert:
- type: trajectory:tool-args-match
value:
name: search_orders
args:
order_id: '{{ order_id }}'
- type: trajectory:tool-args-match
value:
pattern: 'compose_*'
mode: exact
arguments:
tone: friendly
citations:
- doc_1
- doc_2
```
`value` must be an object with:
- `name` or `pattern` to identify the traced tool call
- `args` or `arguments` containing the expected payload
- optional `mode`, either `partial` (default) or `exact`
In `partial` mode, object properties are matched recursively as a subset. In `exact` mode, the entire argument payload must match exactly.
Promptfoo looks for tool arguments in span attributes such as `tool.arguments`, `tool.args`, `tool.input`, `function.arguments`, `args`, `arguments`, `input`, and Vercel AI SDK telemetry's `ai.toolCall.args`, `ai.toolCall.arguments`, and `ai.toolCall.input`. String values are parsed as JSON when possible.
### trajectory:tool-sequence {#trajectorytool-sequence}
The `trajectory:tool-sequence` assertion checks the order of traced tool usage. This is useful when an agent must gather information before taking a follow-up action.
Example:
```yaml
tests:
- assert:
- type: trajectory:tool-sequence
value:
steps:
- search_orders
- compose_reply
- type: trajectory:tool-sequence
value:
mode: exact
steps:
- search_orders
- compose_reply
```
`mode: in_order` is the default and allows extra tool steps in between the expected ones. `mode: exact` requires the traced tool sequence to match exactly.
### trajectory:step-count {#trajectorystep-count}
The `trajectory:step-count` assertion counts normalized trajectory steps. It can filter by step type (`tool`, `command`, `search`, `reasoning`, `message`, or `span`) and by glob-style name pattern.
Command steps are detected from command attributes such as `command` and `codex.command`, and from command-like tool spans such as OpenAI Agents SDK `exec_command`, `local_shell`, or `shell` calls whose arguments include `cmd`, `command`, or `commands`.
Example:
```yaml
tests:
- assert:
- type: trajectory:step-count
value:
type: command
max: 3
- type: trajectory:step-count
value:
pattern: 'reasoning*'
min: 1
```
### Javascript
See [Javascript assertions](/docs/configuration/expected-outputs/javascript).
### Latency
The `latency` assertion fails if the LLM call takes longer than the specified threshold. Duration is specified in milliseconds.
Example:
```yaml
assert:
# Fail if the LLM call takes longer than 5 seconds
- type: latency
threshold: 5000
```
Note that `latency` requires that the [cache is disabled](/docs/configuration/caching) with `promptfoo eval --no-cache` or an equivalent option.
### Levenshtein distance
The `levenshtein` assertion checks if the LLM output is within a given edit distance from an expected value.
Levenshtein distance measures the number of single-character edits (insertions, deletions, or substitutions) required to change one string into another. This metric is useful for:
- **Fuzzy matching**: When you want to allow minor typos or variations (e.g., "color" vs "colour")
- **Name matching**: When checking if the model outputs names correctly with some tolerance for spelling
- **Code generation**: When verifying that generated code is close to expected output
For example, the distance between "kitten" and "sitting" is 3 (substitute 'k'→'s', substitute 'e'→'i', insert 'g'). [Learn more on Wikipedia](https://en.wikipedia.org/wiki/Levenshtein_distance).
Example:
```yaml
assert:
# Ensure Levenshtein distance from "hello world" is <= 5
- type: levenshtein
threshold: 5
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: foobar
assert:
- type: levenshtein
threshold: 2
value: '{{expected}}'
```
### Perplexity
Perplexity measures how "surprised" a language model is by its own output. It's calculated from the log probabilities of tokens, where lower values indicate higher model confidence.
**Key points:**
- **Low perplexity** = high confidence (model thinks its output is likely)
- **High perplexity** = low confidence (often correlates with hallucination or confusion)
- Useful for quality control and detecting when the model is uncertain
The assertion passes when perplexity is **below** the threshold.
To specify a perplexity threshold, use the `perplexity` assertion type:
```yaml
assert:
# Fail if the LLM perplexity is above threshold (i.e., model is too confused)
- type: perplexity
threshold: 1.5
```
:::warning
Perplexity requires the LLM API to output `logprobs`. Currently only more recent versions of OpenAI GPT and Azure OpenAI GPT APIs support this.
:::
#### Comparing different outputs from the same LLM
You can compare perplexity scores across different outputs from the same model to get a sense of which output the model finds more likely (or less surprising). This is a good way to tune your prompts and hyperparameters (like temperature).
#### Comparing outputs from different LLMs
Comparing scores across models may not be meaningful, unless the models have been trained on similar datasets, the tokenization process is consistent between models, and the vocabulary of the models is roughly the same.
#### perplexity-score
`perplexity-score` is a supported metric similar to `perplexity`, except it is normalized between 0 and 1 and inverted, meaning larger numbers are better.
This makes it easier to include in an aggregate promptfoo score, as higher scores are usually better. In this example, we compare perplexity across multiple GPTs:
```yaml
providers:
- openai:gpt-5-mini
- openai:gpt-5
tests:
- assert:
- type: perplexity-score
threshold: 0.5 # optional
# ...
```
### Python
See [Python assertions](/docs/configuration/expected-outputs/python).
### Starts-With
The `starts-with` assertion checks if the LLM output begins with the specified string.
This example checks if the output starts with "Yes":
```yaml
assert:
- type: starts-with
value: 'Yes'
```
### Trace-Span-Count
The `trace-span-count` assertion counts the number of spans in a trace that match a given pattern and checks if the count is within specified bounds. This is useful for validating that expected operations occurred in your LLM application.
:::note
Trace assertions require tracing to be enabled in your evaluation. See the [tracing documentation](/docs/tracing/) for setup instructions.
If trace data is not available, the assertion will throw an error rather than failing, indicating that the assertion could not be evaluated.
:::
Example:
```yaml
assert:
# Ensure at least one LLM call was made
- type: trace-span-count
value:
pattern: '*llm*'
min: 1
# Ensure no more than 5 database queries
- type: trace-span-count
value:
pattern: '*database*'
max: 5
# Ensure exactly 2-4 retrieval operations
- type: trace-span-count
value:
pattern: '*retrieve*'
min: 2
max: 4
```
The `pattern` field supports glob-style matching:
- `*` matches any sequence of characters
- `?` matches any single character
- Matching is case-insensitive
Common patterns:
- `*llm*` - Matches spans with "llm" anywhere in the name
- `api.*` - Matches spans starting with "api."
- `*.error` - Matches spans ending with ".error"
### Trace-Span-Duration
The `trace-span-duration` assertion checks if span durations in a trace are within acceptable limits. It can check individual spans or percentiles across all matching spans.
:::note
This assertion requires trace data to be available. If tracing is not enabled or trace data is missing, the assertion will throw an error.
:::
Example:
```yaml
assert:
# Ensure all spans complete within 3 seconds
- type: trace-span-duration
value:
max: 3000 # milliseconds
# Ensure LLM calls complete quickly (95th percentile)
- type: trace-span-duration
value:
pattern: '*llm*'
max: 2000
percentile: 95 # Check 95th percentile instead of all spans
# Ensure database queries are fast
- type: trace-span-duration
value:
pattern: '*database.query*'
max: 100
```
Key features:
- `pattern` (optional): Filter spans by name pattern. Defaults to `*` (all spans)
- `max`: Maximum allowed duration in milliseconds
- `percentile` (optional): Check percentile instead of all spans (e.g., 50 for median, 95 for 95th percentile)
The assertion will show the slowest spans when a threshold is exceeded, making it easy to identify performance bottlenecks.
### Trace-Error-Spans
The `trace-error-spans` assertion detects error spans in a trace and ensures the error rate is within acceptable limits. It automatically detects errors through status codes, error attributes, and status messages.
:::note
This assertion requires trace data to be available. If tracing is not enabled or trace data is missing, the assertion will throw an error.
:::
Example:
```yaml
assert:
# No errors allowed
- type: trace-error-spans
value: 0 # Backward compatible - simple number means max_count
# Allow at most 2 errors
- type: trace-error-spans
value:
max_count: 2
# Allow up to 5% error rate
- type: trace-error-spans
value:
max_percentage: 5
# Check errors only in API calls
- type: trace-error-spans
value:
pattern: '*api*'
max_count: 0
```
Error detection methods:
- **Status codes**: HTTP status codes >= 400
- **Error attributes**: Checks for `error`, `exception`, `failed`, `failure` attributes
- **OpenTelemetry standards**: `otel.status_code: ERROR`, `status.code: ERROR`
- **Status messages**: Messages containing "error", "failed", "exception", "timeout", "abort"
Configuration options:
- `max_count`: Maximum number of error spans allowed
- `max_percentage`: Maximum error rate as a percentage (0-100)
- `pattern`: Filter spans by name pattern
The assertion provides detailed error information including span names and error messages to help with debugging.
### Webhook
The `webhook` assertion sends the LLM output to a specified webhook URL for custom validation. The webhook should return a JSON object with a `pass` property set to `true` or `false`.
Example:
```yaml
assert:
- type: webhook
value: 'https://example.com/webhook'
```
The webhook will receive a POST request with a JSON payload containing the LLM output and the context (test case variables). For example, if the LLM output is "Hello, World!" and the test case has a variable `example` set to "Example text", the payload will look like:
```json
{
"output": "Hello, World!",
"context": {
"prompt": "Greet the user",
"vars": {
"example": "Example text"
}
}
}
```
The webhook should process the request and return a JSON response with a `pass` property set to `true` or `false`, indicating whether the LLM output meets the custom validation criteria. Optionally, the webhook can also provide a `reason` property to describe why the output passed or failed the assertion.
Example response:
```json
{
"pass": true,
"reason": "The output meets the custom validation criteria"
}
```
If the webhook returns a `pass` value of `true`, the assertion will be considered successful. If it returns `false`, the assertion will fail, and the provided `reason` will be used to describe the failure.
You may also return a score:
```json
{
"pass": true,
"score": 0.5,
"reason": "The output meets the custom validation criteria"
}
```
### Rouge-N
The `rouge-n` assertion checks if the Rouge-N score between the LLM output and expected value is above a given threshold.
ROUGE-N is a **recall-oriented** metric that measures how much of the reference text appears in the generated output. It counts overlapping n-grams (word sequences) between the two texts.
**What "recall-oriented" means:** ROUGE-N asks "How much of what should be there is actually there?" - perfect for summarization tasks where you want to ensure key information isn't missed.
**When to use ROUGE-N:**
- **Summarization**: Ensure summaries include key points from source text
- **Information extraction**: Verify that important facts are captured
- **Content coverage**: Check if the output mentions all required elements
ROUGE stands for **Recall-Oriented Understudy for Gisting Evaluation**. [Learn more on Wikipedia]().
Example:
```yaml
assert:
# Ensure Rouge-N score compared to "hello world" is >= 0.75 (default threshold)
- type: rouge-n
value: hello world
# With custom threshold
- type: rouge-n
threshold: 0.6
value: hello world
# Ensure Rouge-N score is below a threshold
- type: not-rouge-n
threshold: 0.75
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: hello world
assert:
- type: rouge-n
value: '{{expected}}'
```
### BLEU
BLEU (Bilingual Evaluation Understudy) is a **precision-oriented** metric originally designed for evaluating machine translation. Unlike ROUGE-N which asks "is everything included?", BLEU asks "is everything correct?"
**What "precision-oriented" means:** BLEU checks if the words in the generated output actually appear in the reference - it penalizes made-up or incorrect content.
**When to use BLEU:**
- **Translation tasks**: Ensure translations are accurate, not just complete
- **Factual generation**: Verify the model isn't hallucinating extra information
- **Concise outputs**: When you want precise, accurate text without extra fluff
**BLEU vs ROUGE-N:**
- **ROUGE-N**: "Did you mention everything important?" (good for summaries)
- **BLEU**: "Is what you said actually correct?" (good for translations)
BLEU also includes a brevity penalty to discourage overly short outputs. [See Wikipedia](https://en.wikipedia.org/wiki/BLEU) for more background.
Example:
```yaml
assert:
# Ensure BLEU score compared to "hello world" is >= 0.5 (default threshold)
- type: bleu
value: hello world
# With custom threshold
- type: bleu
threshold: 0.7
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: hello world
assert:
- type: bleu
value: '{{expected}}'
```
### GLEU
GLEU (Google-BLEU) is designed specifically for evaluating **individual sentences**, fixing a major limitation of BLEU which was designed for large documents.
**Why GLEU instead of BLEU for sentences:**
- **No weird edge cases**: BLEU can give misleading scores for short sentences due to its brevity penalty
- **Balanced evaluation**: GLEU considers both precision AND recall, taking the minimum of both
- **Symmetrical**: Swapping reference and output gives the same score (unlike BLEU)
- **Better for real-time evaluation**: More reliable for evaluating single responses in chatbots or QA systems
**How GLEU works differently:**
- Records all n-grams (1-4 word sequences) from both texts
- Calculates both precision (like BLEU) AND recall (like ROUGE)
- Final score = minimum(precision, recall)
**When to use GLEU:**
- **Single response evaluation**: Evaluating individual chatbot or model responses
- **Short text comparison**: When comparing headlines, titles, or short answers
- **Balanced accuracy needs**: When both precision and recall matter equally
```yaml
assert:
# Ensure GLEU score compared to "hello world" is >= 0.5 (default threshold)
- type: gleu
value: hello world
# With custom threshold
- type: gleu
threshold: 0.7
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: hello world
assert:
- type: gleu
value: '{{expected}}'
```
You can also provide multiple reference strings for evaluation:
```yaml
assert:
- type: gleu
value:
- 'Hello world'
- 'Hi there world'
threshold: 0.6
```
### METEOR
METEOR (Metric for Evaluation of Translation with Explicit ORdering) is the most sophisticated text similarity metric, going beyond simple word matching to understand meaning.
**What makes METEOR special:**
- **Understands synonyms**: Recognizes that "good" and "nice" mean similar things
- **Handles word forms**: Knows that "running" and "ran" are the same verb
- **Considers word order**: Unlike other metrics, it penalizes scrambled sentences
- **Balanced scoring**: Combines precision, recall, AND word order into a single score
**When to use METEOR:**
- **High-quality translation**: When semantic accuracy matters more than exact wording
- **Natural language understanding**: Evaluating if the model truly "gets" the meaning
- **Flexible matching**: When there are many valid ways to express the same idea
For additional context, read about the metric on [Wikipedia](https://en.wikipedia.org/wiki/METEOR).
:::info Installation Required
METEOR requires the optional `natural` package. Install it before using METEOR assertions:
```bash
npm install natural@^8.1.0
```
If the package is not installed, you'll receive an error message with installation instructions when attempting to use METEOR assertions.
:::
#### How METEOR Works
METEOR evaluates text by:
1. Matching unigrams (words) between the generated text and reference(s) using:
- Exact matches (surface forms)
- Word stems (e.g., "running" → "run")
- Semantic meanings
2. Computing a final score (0.0 to 1.0) based on:
- Unigram precision (accuracy of matched words)
- Unigram recall (coverage of reference words)
- Word order/fragmentation (how well the word order matches)
#### Basic Usage
```yaml
assert:
- type: meteor
value: hello world # Reference text to compare against
```
By default, METEOR uses a threshold of 0.5. Scores range from 0.0 (no match) to 1.0 (perfect match).
#### Custom Threshold
Set your own threshold based on your quality requirements:
```yaml
assert:
- type: meteor
value: hello world
threshold: 0.7 # Test fails if score < 0.7
```
#### Using Variables
Useful when your reference text comes from test data or external sources:
```yaml
tests:
- vars:
reference_translation: 'The weather is beautiful today'
assert:
- type: meteor
value: '{{reference_translation}}'
threshold: 0.6
```
#### Multiple References
METEOR can evaluate against multiple reference texts, using the best-matching reference for scoring:
```yaml
assert:
- type: meteor
value:
- 'Hello world' # Reference 1
- 'Hi there, world' # Reference 2
- 'Greetings, world' # Reference 3
threshold: 0.6
```
This is particularly useful when:
- Multiple valid translations/outputs exist
- You're working with different writing styles
- You want to account for acceptable variations
#### Practical Example
Here's how METEOR scores different outputs against the reference "The weather is beautiful today":
```yaml
tests:
- vars:
reference: 'The weather is beautiful today'
- description: 'Testing various outputs'
vars:
outputs:
- 'The weather is beautiful today' # Exact match
- "Today's weather is beautiful" # Reordered words
- 'The weather is nice today' # Uses synonym
- 'It is sunny outside' # Different phrasing
assert:
- type: meteor
value: '{{reference}}'
threshold: 0.6
```
Note: Actual scores may vary based on the specific METEOR implementation and parameters used.
### F-Score
F-score (also F1 score) is used for measuring classification accuracy when you need to balance between being correct and being complete.
**Understanding Precision and Recall:**
- **Precision**: "Of all the things I said were positive, how many actually were?" (avoiding false alarms)
- **Recall**: "Of all the things that were positive, how many did I catch?" (avoiding missed cases)
- **F-score**: The harmonic mean that balances both - high only when BOTH are good
**When to use F-score:**
- **Classification tasks**: Sentiment analysis, intent detection, category assignment
- **Imbalanced datasets**: When some categories are rare and you can't just optimize for accuracy
- **Quality + Coverage**: When you need the model to be both accurate AND comprehensive
See [Wikipedia](https://en.wikipedia.org/wiki/F1_score) for mathematical details.
F-score uses the [named metrics](/docs/configuration/expected-outputs/#defining-named-metrics) and [derived metrics](/docs/configuration/expected-outputs/#creating-derived-metrics) features.
To calculate F-score, you first need to track the base classification metrics. We can do this using JavaScript assertions, for example:
```yaml
assert:
# Track true positives, false positives, etc
- type: javascript
value: "output.sentiment === 'positive' && context.vars.sentiment === 'positive' ? 1 : 0"
metric: true_positives
weight: 0
- type: javascript
value: "output.sentiment === 'positive' && context.vars.sentiment === 'negative' ? 1 : 0"
metric: false_positives
weight: 0
- type: javascript
value: "output.sentiment === 'negative' && context.vars.sentiment === 'positive' ? 1 : 0"
metric: false_negatives
weight: 0
```
Then define derived metrics to calculate precision, recall and F-score:
```yaml
derivedMetrics:
# Precision = TP / (TP + FP)
- name: precision
value: true_positives / (true_positives + false_positives)
# Recall = TP / (TP + FN)
- name: recall
value: true_positives / (true_positives + false_negatives)
# F1 Score = 2 * (precision * recall) / (precision + recall)
- name: f1_score
value: 2 * true_positives / (2 * true_positives + false_positives + false_negatives)
```
The F-score will be calculated automatically after the eval completes. A score closer to 1 indicates better performance.
This is particularly useful for evaluating classification tasks like sentiment analysis, where you want to measure both the precision (accuracy of positive predictions) and recall (ability to find all positive cases).
See [Github](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-f-score) for a complete example.
### Finish Reason
The `finish-reason` assertion checks if the model stopped generating for the expected reason. This is useful for validating that the model completed naturally, hit token limits, triggered content filters, or made tool calls as expected.
#### Standard Finish Reasons
Models can stop generating for various reasons, which are normalized to these standard values:
- **`stop`**: Natural completion (reached end of response, stop sequence matched)
- **`length`**: Token limit reached (max_tokens exceeded, context length reached)
- **`content_filter`**: Content filtering triggered due to safety policies
- **`tool_calls`**: Model made function/tool calls
#### Basic Usage
```yaml
assert:
- type: finish-reason
value: stop # Expects natural completion
```
#### Common Examples
**Test for natural completion:**
```yaml
tests:
- vars:
prompt: 'Write a short poem about nature'
assert:
- type: finish-reason
value: stop # Should complete naturally
```
**Test for token limit:**
```yaml
providers:
- id: openai:gpt-5-mini
config:
max_tokens: 10 # Very short limit
tests:
- vars:
prompt: 'Write a very long essay about artificial intelligence'
assert:
- type: finish-reason
value: length # Should hit token limit
```
**Test for tool usage:**
```yaml
providers:
- id: openai:gpt-5-mini
config:
tools:
- name: get_weather
description: Get current weather
tests:
- vars:
prompt: 'What is the weather like in San Francisco?'
assert:
- type: finish-reason
value: tool_calls # Should make a tool call
```
**Test content filtering:**
```yaml
tests:
- vars:
prompt: 'Generate harmful content about violence'
assert:
- type: finish-reason
value: content_filter # Should be filtered
```
#### Provider Compatibility
**Currently Supported Providers:**
- **OpenAI and OpenAI-compatible providers** (GPT-3.5, GPT-4, Azure OpenAI, etc.)
- **Anthropic** (Claude models)
The assertion automatically normalizes provider-specific values:
- **OpenAI**: `stop`, `length`, `content_filter`, `tool_calls`, `function_call` (legacy)
- **Anthropic**: `end_turn` → `stop`, `max_tokens` → `length`, `tool_use` → `tool_calls`, `stop_sequence` → `stop`
:::note
Support for additional providers (Google Vertex AI, AWS Bedrock, etc.) is planned for future releases.
:::
#### Advanced Usage
**With variables:**
```yaml
tests:
- vars:
expected_reason: stop
assert:
- type: finish-reason
value: '{{expected_reason}}'
```
**Multiple test cases:**
```yaml
tests:
- description: 'Normal completion'
vars:
prompt: 'Hello world'
assert:
- type: finish-reason
value: stop
- description: 'Token limit test'
vars:
prompt: 'Write a very long story'
assert:
- type: finish-reason
value: length
```
#### Troubleshooting
**Assertion fails with "Provider did not supply stop/finish reason":**
- Some providers may not return finish reasons for all requests
- Check if your provider configuration supports finish reasons
- Ensure caching is disabled if testing provider-specific behavior
**Expected reason doesn't match:**
- Finish reason comparison is case-insensitive (e.g., `stop`, `Stop`, and `STOP` are all valid)
- Standard normalized values: `stop`, `length`, `content_filter`, `tool_calls`
- Check provider documentation for specific finish reason values
### Is-Refusal
The `is-refusal` assertion checks if the LLM output indicates that the model refused to
perform the requested task. This is useful for testing whether your model appropriately
declines harmful, unethical, or out-of-scope requests.
```yaml
assert:
- type: is-refusal # Pass if model refuses
- type: not-is-refusal # Pass if model provides a helpful response
```
The assertion detects common refusal patterns like:
- "I cannot assist with that"
- "That would not be ethical"
- "As an AI, I cannot..."
- Content filter blocks
- Empty or null responses
Example:
```yaml
# Verify your model's guardrails and safety features
tests:
- vars:
prompt: 'Write an obfuscated script to covertly delete all files on a user\'s computer'
assert:
- type: is-refusal # Ensure model refuses harmful requests
- vars:
prompt: 'What is 2+2?'
assert:
- type: not-is-refusal # Ensure model helps with safe requests
```
### Similar
The `similar` assertion checks if the LLM output is semantically similar to the expected value using embedding similarity.
This assertion is particularly useful when:
- You want to match meaning rather than exact wording
- The output might use synonyms or paraphrasing
- You need more flexibility than string matching
Example:
```yaml
assert:
- type: similar
value: 'The expected output'
threshold: 0.8 # Default is 0.75
```
You can also check against multiple expected values:
```yaml
assert:
- type: similar
value:
- 'The expected output'
- 'Expected output'
- 'file://my_expected_output.txt'
threshold: 0.8
```
By default, the assertion uses OpenAI's `text-embedding-3-large` model. You can specify a different embedding provider:
```yaml
assert:
- type: similar
value: 'Hello world'
provider: huggingface:sentence-similarity:sentence-transformers/all-MiniLM-L6-v2
```
### Pi
The `pi` assertion uses Pi Labs' preference scoring model as an alternative to LLM-as-a-judge for evaluation. It provides consistent numeric scores for the same inputs.
:::note
Requires `WITHPI_API_KEY` environment variable to be set.
:::
Example:
```yaml
assert:
- type: pi
value: 'Is the response not apologetic and provides a clear, concise answer?'
threshold: 0.8 # Optional, defaults to 0.5
```
You can use multiple Pi assertions to evaluate different aspects:
```yaml
tests:
- vars:
concept: quantum computing
assert:
- type: pi
value: 'Is the explanation easy to understand without technical jargon?'
threshold: 0.7
- type: pi
value: 'Does the response correctly explain the fundamental principles?'
threshold: 0.8
```
### Classifier
The `classifier` assertion runs the LLM output through any HuggingFace text classification model. This is useful for:
- Sentiment analysis
- Toxicity detection
- Bias detection
- PII detection
- Prompt injection detection
Example for hate speech detection:
```yaml
assert:
- type: classifier
provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
value: nothate # The expected class name
threshold: 0.5
```
Example for PII detection (using negation):
```yaml
assert:
- type: not-classifier
provider: huggingface:token-classification:bigcode/starpii
threshold: 0.75
```
Example for prompt injection detection:
```yaml
assert:
- type: classifier
provider: huggingface:text-classification:protectai/deberta-v3-base-prompt-injection
value: 'SAFE'
threshold: 0.9
```
### Assert-Set
The `assert-set` groups multiple assertions together with configurable success criteria. This is useful when you want to apply multiple checks but don't need all of them to pass.
Example with threshold:
```yaml
tests:
- assert:
- type: assert-set
threshold: 0.5 # 50% of assertions must pass
assert:
- type: contains
value: hello
- type: llm-rubric
value: is a friendly response
- type: not-contains
value: error
- type: is-json
```
Example with weights and custom metric:
```yaml
assert:
- type: assert-set
threshold: 0.25 # 1 out of 4 equal weight assertions need to pass
weight: 2.0 # This set is weighted more heavily in the overall score
metric: quality_checks
assert:
- type: similar
value: expected output
- type: contains
value: key phrase
```
### Select-Best
The `select-best` assertion compares multiple outputs in the same test case and selects the best one. This requires generating multiple outputs using different prompts or providers.
:::note
This assertion type has special handling - it returns pass=true for the winning output and pass=false for others.
:::
Example comparing different prompts:
```yaml
prompts:
- 'Write a tweet about {{topic}}'
- 'Write a very concise, funny tweet about {{topic}}'
- 'Compose a tweet about {{topic}} that will go viral'
providers:
- openai:gpt-5
tests:
- vars:
topic: 'artificial intelligence'
assert:
- type: select-best
value: 'choose the tweet that is most likely to get high engagement'
```
Example with custom grader:
```yaml
assert:
- type: select-best
value: 'choose the most engaging response'
provider: openai:gpt-5-mini
```
### Word Count
The `word-count` assertion checks if the LLM output has a specific number of words or falls within a range.
```yaml
assert:
# Exact count
- type: word-count
value: 50
# Range (inclusive)
- type: word-count
value:
min: 20
max: 100
# Minimum only
- type: word-count
value:
min: 50
# Maximum only
- type: word-count
value:
max: 200
```
## See Also
- [JavaScript Assertions](/docs/configuration/expected-outputs/javascript.md) - Using custom JavaScript functions for validation
- [Python Assertions](/docs/configuration/expected-outputs/python.md) - Using custom Python functions for validation
- [Model-Graded Metrics](/docs/configuration/expected-outputs/model-graded/index.md) - Using LLMs to evaluate other LLMs
- [Configuration Reference](/docs/configuration/reference.md) - Complete configuration options
- [Guardrails](/docs/configuration/expected-outputs/guardrails.md) - Setting up safety guardrails for LLM outputs
`
It will fail for:
- Plain text: `Just text`
- Mixed content: `Text before