Evaluations
Evaluations automatically assess the quality of your LLM generations. PostHog supports three types of evaluations:
- LLM-as-a-judge – Uses an LLM to score each generation against a prompt you define, returning a pass/fail result, a numeric score, or categories. You choose a judge method: LLM with explanation provides written reasoning, while Decision model returns typed results without reasoning. Great for nuanced, subjective checks like tone, helpfulness, or hallucination detection.
- Code-based (Hog) – Runs deterministic code you write against each generation, returning a pass/fail result, a numeric score, or categories. Great for rule-based checks like format validation, keyword detection, or length limits. Free to run with no LLM cost.
- Sentiment analysis – Classifies the sentiment of user messages as positive, neutral, or negative using a local ML model. Great for monitoring how users feel during conversations with your AI features. Free to run with no LLM cost.
Choose online or offline evals
PostHog runs two kinds of evals, and most teams use both:
- Online evals – Score your live production traffic in PostHog as it arrives. Use them to find the problems your users hit. This page covers online evals.
- Offline evals – Score a fixed set of test cases in your CI before you ship a change. Use them to stop known problems from coming back. See offline evals.
To run an online eval over traffic you captured before you created it, use a backfill.
Why use evaluations?
- Monitor output quality at scale – Automatically check if generations are helpful, relevant, or safe without manual review.
- Detect problematic content – Catch hallucinations, toxicity, or jailbreak attempts before they reach users.
- Track quality trends – See pass rates across models, prompts, or user segments over time.
- Debug with reasoning – LLM judges using the LLM with explanation method provide written reasoning, and Hog evaluations can include reasoning through
print(). Decision model judges return results without written reasoning.
Choosing an evaluation type
| LLM-as-a-judge | Code-based (Hog) | Sentiment analysis | |
|---|---|---|---|
| Best for | Subjective quality checks (tone, helpfulness, hallucination) | Deterministic rule-based checks (format, keywords, length) | Tracking how users feel during conversations |
| Cost | LLM API call per evaluation | Free | Free |
| Speed | Seconds | Milliseconds | Milliseconds |
| Output | Pass/fail, numeric score, or categories; written reasoning with LLM with explanation, no reasoning with Decision model | Pass/fail, numeric score, or categories; optional reasoning from print() | Positive, neutral, or negative with a confidence score |
| Setup | Write a prompt | Write Hog code | Pick the type – no prompt or code required |
LLM with explanation, decision models, and Hog code each judge differently, cost differently, and return different results. Compare them in Choose a judge method before you create an evaluation.
Choosing what to evaluate
LLM-as-a-judge and code-based evaluations have an Evaluate setting that picks what one run grades:
- Each generation grades every matching generation on its own, right after it's ingested.
- Whole trace grades all the events in a trace together, after the trace settles.
- Whole session grades every trace in a session together, after the session settles.
Property filters still match individual generations. A trace or session is evaluated when any of its generations match, and the sampling rate applies to the whole trace or session.
Whole session only runs on generations that carry the $ai_session_id property. If your events don't include it, a session evaluation has nothing to grade and never produces results. See how to set a session ID, or pick Whole trace or Each generation instead.
Sentiment evaluations always grade each generation.
Choosing an output type
LLM-as-a-judge and code-based evaluations have an Output type setting. You pick it when you create the evaluation, and you can't change it later.
- Boolean returns pass or fail. Use it for yes/no checks, like "the response is valid JSON".
- Numeric score returns a number. Use it for quality ratings, like a 1 to 5 helpfulness score, or for measured values, like latency or response length.
- Categorical returns one or more labels from a list you define. Use it to sort results into failure modes, like "wrong tool", "hallucination", or "off topic".
Numeric scores
You can set a Minimum and Maximum score. A score outside these bounds is skipped, not graded. Step suggests the score increment. PostHog asks an LLM judge to use it, but stores the score exactly as returned, so a judge can still return a value like 3.7 when the step is 1.
To get pass rates and reports, turn on Set a passing rule and choose At least (≥) or At most (≤) a threshold. Without a passing rule, you still get the scores, but no pass rates or reports. If you change the rule later, PostHog applies the new rule to past results too.
Categories
Add the categories the evaluation can return. Each category has a key, which PostHog stores, and a label, which PostHog displays. Keys use lowercase letters, numbers, _, and -. Keep a key the same when you rename its label, so past results keep their meaning. An evaluation can have up to 100 categories.
Categories per result controls how many categories one result can have:
- One category returns exactly one category.
- Multiple categories returns any number of categories, including none.
To get pass rates and reports, choose Passing categories. A result passes only when all of its categories are passing categories. For example, if resolved is the only passing category:
resolvedpasses.resolvedandescalatedfails, becauseescalatedisn't a passing category.- A result with no categories fails.
With Multiple categories, you can also leave Passing categories empty. Then a result passes only when it has no categories. Use this when every category is a problem, like hallucination or wrong_tool, so a good result has none.
If you change the passing categories later, PostHog applies them to past results too.
Not applicable results
Turn on Allow N/A responses for a boolean, numeric, or categorical evaluation when its criteria don't apply to every generation. Both LLM judges and Hog evaluations support N/A. N/A results are excluded from pass/fail grading and pass rates.
For LLM judges, describe when the criteria don't apply in your prompt. For Hog evaluations, return null to mark a result as N/A when Allow N/A responses is enabled.
With Multiple categories, an empty list ([]) is a valid, applicable result that means no categories match. Its passing rule still determines whether it passes. It is different from null, which marks the result as N/A when enabled.
LLM-as-a-judge evaluations
To set up an LLM judge, including templates, reasoning levels, and prompt tips, see Choose a judge method.
For each topic, see What the judge receives, Tool call and result correlation, Tool catalog format, Shaping custom events for evaluations, Built-in templates, Explanation or decision model, Setting a reasoning level, Creating an LLM judge evaluation, and Writing custom prompts.
Code-based evaluations (Hog)
To write a code-based evaluation in Hog, see Choose a judge method.
For each topic, see Available globals, Creating a code-based evaluation, Writing Hog evaluation code, Testing on sample data, and Using AI to generate evaluations.
Sentiment analysis evaluations
Sentiment evaluations classify the sentiment of the user messages in each generation as positive, neutral, or negative. They're ideal for monitoring how users feel during conversations with your AI features – spotting frustrated users, tracking satisfaction trends, and prioritizing the conversations that matter most.
How they work
When a generation is sampled, a local ML model classifies each user message in it as positive, neutral, or negative. The model runs inside PostHog, so no data goes to a third-party AI provider and there's no LLM cost. See how sentiment classification works for the model and what it reads.
Sentiment evaluations share the same sampling rate and property filter options as the other evaluation types, so you can scope them to production traffic, specific models, or particular user segments.
Creating a sentiment evaluation
- Navigate to AI Evals > Evaluations
- Click New evaluation
- Select Sentiment analysis as the evaluation type
- Configure:
- Name: A descriptive name for the evaluation
- Sampling rate: Percentage of generations to evaluate (0.1% – 100%)
- Property filters (optional): Narrow which generations to evaluate using event or person properties
- Enable the evaluation and click Save
Viewing sentiment results
Once a sentiment evaluation is running, the Sentiment tab in AI Observability lets you browse user messages by sentiment, and traces and generations display a color-coded sentiment bar. See Sentiment classification for details on the viewing experience.
Test and refine your eval
Check a new eval's verdicts against your own before you rely on it.
- Try it on real traffic. For a code-based evaluation, click Test on sample to run it on recent generations without saving results. For an LLM judge, start with a low sampling rate, or backfill a day of traffic.
- Check its verdicts by hand. Read a sample of results, or score the same traces yourself with Trace Reviews. Note false positives (good responses it failed) and false negatives (bad responses it passed).
- Refine it. Make the pass and fail criteria more specific, add the cases it got wrong to your prompt as examples, or change the output type or judge method.
- Repeat until its verdicts match yours. Then raise the sampling rate and let its report summarize results for you.
Viewing results
The Evaluations page shows all your evaluations with their pass rates and recent activity. Click an evaluation to see its run history, including individual results and any reasoning from the evaluation.
You can also filter generations by evaluation results or create insights based on evaluation data to build quality monitoring dashboards.
Catch regressions with datasets
When an eval flags a bad response, add that trace to a dataset. Your CI can then run the dataset with offline evals on every change, so the same problem can't ship again unnoticed. To see the full loop, read turn a production failure into a test case.
Evaluating past data
To run an evaluation over data you captured before you created it, see Backfills.
For each topic, see How backfills work, Creating a backfill, Re-evaluating generations that already have a result, Reading backfill progress, When the count is zero, and Backfill pricing.
Evaluation reports
To get evaluation results summarized in email or Slack, see Evaluation reports.
For each topic, see Default report configuration, Report metrics by evaluation type, and Customizing reports.
Managing evaluations via MCP
You can also manage evaluations through the PostHog MCP server using AI agents like Claude Code, Cursor, or any MCP-connected tool.
The MCP server provides these evaluation management tools:
| Tool | Description |
|---|---|
llma-evaluation-list | List evaluations with optional search and enabled filter |
llma-evaluation-get | Get a specific evaluation by UUID |
llma-evaluation-create | Create LLM judge or Hog code evaluations |
llma-evaluation-update | Update evaluation config, name, or enabled status |
llma-evaluation-delete | Delete an evaluation (soft delete) |
llma-evaluation-run | Trigger an async evaluation run on a specific generation event |
This enables teams to manage evaluations programmatically from agent workflows without using the web UI.
Evaluation status and errors
Each evaluation has a status that indicates whether it's running:
- Active – The evaluation is enabled and running on sampled generations.
- Paused – You disabled the evaluation. It won't run until you re-enable it.
- Error – PostHog automatically disabled the evaluation due to an issue that needs attention.
Automatic error states
PostHog automatically disables evaluations and sets them to error status when:
| Error reason | Cause | How to resolve |
|---|---|---|
| No provider API key configured | The evaluation has no provider API key to run with. | Add a provider API key in Settings > AI Observability. |
| Provider key deleted | The provider API key used by this evaluation was removed. | Add a new provider API key in Settings > AI Observability. |
Identifying errored evaluations
On the Evaluations list page, errored evaluations display a red Error pill instead of the enable/disable toggle. Hover over the pill to see the specific error reason. Errored evaluations sort to the top of the list.
On the evaluation detail page, the header shows a red Error tag and a banner explains the cause with recovery steps.
Re-enabling errored evaluations
You must resolve the underlying issue before re-enabling an errored evaluation:
- No provider API key configured – Add a provider API key.
- Provider key deleted – Add a new provider API key.
Once the issue is resolved, open the evaluation and toggle it back on.
Pricing
Each evaluation run counts as one AI Observability event toward your quota.
LLM judge evaluations use an LLM to score your generations and require a provider API key. Add your own key from OpenAI, Azure OpenAI, Google Gemini, Anthropic, OpenRouter, Fireworks, or Together AI in Settings > AI Observability to enable LLM-judge evaluations.
If a provider API key is missing or deleted, PostHog automatically disables the affected evaluations and shows an error status. See Evaluation status and errors for details on error states and how to resolve them.
Code-based evaluations have no LLM cost – they run your Hog code directly with no external API calls.
Sentiment evaluations have no LLM cost – they run a local ML model with no external API calls.
Use sampling rates strategically to balance coverage and cost – 5-10% sampling often provides sufficient signal for quality monitoring.
Backfills bill the same way. See Backfill pricing for how to check the count before starting one.





