Datasets

Datasets are in beta

Datasets are currently in beta. We'd love your feedback as we develop the feature.

Datasets let you curate sets of inputs and expected outputs that you can replay against prompt or model changes to catch regressions before they reach production. Instead of hoping a prompt tweak didn't break anything, you keep a growing collection of the cases you care about – the tricky ones, the ones users complained about, the ones that broke last time – and re-run them.

Every change is versioned, so you can see what a dataset looked like when you ran a test. Revisions are available until they expire under the retention policy.

Why use datasets?

  • Catch regressions before shipping – Replay real cases against a new prompt, model, or agent version.
  • Turn production failures into test cases – Add a trace to a dataset directly from AI observability the moment you spot a problem.
  • Keep tests reproducible – Every dataset change creates an immutable revision, so a test run can be pinned to exactly the data it used.
  • Work programmatically – Build and update datasets from agents and scripts, not just the PostHog UI.

How datasets work

Datasets have three layers:

  • Dataset – A named collection of items, scoped to a project. It has a description and a free-form metadata object.
  • Dataset item – A single test case with a stable ID. An item's content lives in its versions, not on the item itself.
  • Item version – An immutable snapshot of one item's content. Editing an item never overwrites anything, it appends a new version.

On top of item versions sit dataset revisions: monotonically increasing snapshots of the whole dataset. Every item mutation creates a new revision, so revision 12 describes exactly which version of every item was current at that point.

A brand new dataset has no revision. Its first revision is created along with its first item.

Item fields

FieldDescription
inputThe input given to the system under test. Any non-null JSON value – a string, an array of messages, an object.
expected_outputOptional. The output you want, authored by you. This is what you compare against.
source_outputOptional. The actual output captured from the source trace, kept for reference.
metadataOptional JSON object for your own labels – category, difficulty, owning team, whatever you filter on later.
client_item_idOptional case-sensitive key you own. Makes creates idempotent and safe to retry. It can't be changed.
source_trace_idThe trace the item came from. Provide it together with source_timestamp.
source_event_idOptional. The specific event within the trace.
source_timestampRequired when you set source_trace_id. Needed to look the source trace back up.

Both expected_output and source_output are optional, so you can save a case you know is wrong before you've decided what the right answer looks like.

Creating a dataset

You can create a dataset in the PostHog UI or with the PostHog MCP server.

Adding items

From a trace

The fastest way to build a useful dataset is to harvest real traffic. On a trace, click Add to dataset and pick a dataset. PostHog pre-fills the item with the trace's input, its output as source_output, and the trace provenance fields, so you can jump back to where the case came from.

Manually

Open a dataset and add an item directly, filling in Input, Expected output, and Metadata as JSON.

Example: turn a production failure into a test case

Twig's AI trip planner helps guests find a place to stay. One day it books a group of four into a nest that sleeps two. Here's how that failure becomes a test that runs on every change:

  1. An online evaluation called "Stay fits the group" fails the trace in production.
  2. On the trace, click Add to dataset and pick your regression dataset. PostHog fills in the input and saves the planner's answer as source_output.
  3. Open the new item and set Expected output to a stay that sleeps four.
  4. Fix the guest filter in the planner and open a pull request.
  5. Your CI runs offline evals against the latest dataset revision, new item included. The fix passes, so you merge it.
  6. Every later pull request runs the same item. If the bug comes back, CI fails before a guest sees it.

Versioning

Every item change creates an immutable version and a new dataset revision. You can view past revisions and restore an earlier version of an item while that history is still available under the retention policy.

Archiving

Datasets and items can be archived instead of deleted immediately. An archived dataset stays readable until it expires, but rejects item mutations until you restore it. Archived items are filtered out of the default view – and out of exports. Their history follows the retention policy below.

Data retention

PostHog deletes data after these periods without use:

  • Active datasets: 90 days, starting at creation.
  • Archived datasets: 30 days, starting when archived.
  • Older revisions: 30 days, starting when replaced by a newer revision.

Reading contents in PostHog or through the SDK/API, exporting, editing, restoring, or using them in an evaluation counts as use. Listing datasets or revisions does not.

Using a revision restarts the clock for that revision and its dataset, but not other revisions. The latest revision stays available with its dataset. Ongoing evaluation uploads and exports keep the data they need until they finish.

Expiring soon appears in the final 7 days. Expired datasets and revisions cannot be read or restored. Export and download anything you need to keep.

Historical evaluation scores and scorer configurations are kept. Evaluation inputs, outputs, and other payloads follow a separate 30-day policy.

Exporting a dataset

Exports let you run a dataset through your own harness, in CI or on your machine, and keep the results comparable over time.

Click Export on a dataset to generate a JSONL file. Exports are:

  • Asynchronous – The export is prepared in the background, then made available to download. Exported files expire, so download or re-export as needed.
  • Pinned to a revision – Exporting from Latest pins to the current revision, and exporting while viewing a historical revision pins to that one. Because revisions are immutable, re-downloading the same export always gives you identical data.
  • Active items only – Archived items are excluded.

Each line is one item, with the revision recorded on every row:

JSON
{"dataset_id":"018f...","dataset_revision":12,"item_id":"018f...","client_item_id":"ticket-4821","version":3,"input":{"messages":[{"role":"user","content":"How do I cancel?"}]},"expected_output":"Point the user to the billing settings page.","source_output":"You can cancel any time.","metadata":{"category":"billing"},"source_trace_id":"trace_abc","source_event_id":"event_def","source_timestamp":"2026-05-04T10:12:33+00:00"}

Pinning to a revision is what makes an export worth storing next to your test results: "this run scored 84% on revision 12" stays meaningful even after the dataset has moved on.

Programmatic access

MCP

The PostHog MCP server exposes datasets to AI agents like Claude Code and Cursor, so an agent that just debugged a bad generation can add it to your regression set without you switching windows.

ToolDescription
llma-dataset-listList datasets, with search and an active/archived filter
llma-dataset-getGet a dataset by ID
llma-dataset-createCreate a dataset
llma-dataset-updateUpdate a dataset's name, description, or metadata
llma-dataset-archiveArchive a dataset
llma-dataset-restoreRestore an archived dataset
llma-dataset-revision-listList a dataset's revisions, newest first
llma-dataset-item-listList items, optionally as they appeared at a specific revision
llma-dataset-item-getGet an item, optionally at a specific revision
llma-dataset-item-createAdd an item and create its first version
llma-dataset-item-updateCreate a new version of an item
llma-dataset-item-archiveArchive an item
llma-dataset-item-restoreRestore an archived item, optionally from a chosen historical version
llma-dataset-item-version-listList an item's version history

Pass a client_item_id when an agent creates items. An identical retry returns the existing item instead of creating a duplicate, which makes agent loops safe to re-run.

API

The same operations are available over REST at /api/projects/:project_id/datasets and /api/projects/:project_id/dataset_items, including creating exports and downloading their content. Use a personal API key with the dataset scope.

CLI

CLI access is on the way, so you'll be able to pull a dataset revision and run it as part of a build without writing API glue. It isn't available yet.

Limits

These limits are subject to change.

ResourceLimitWhat counts
Datasets per project100 by defaultActive datasets and archived datasets that haven't expired.
Items per dataset2,000Both active and archived items.
Retained versions per item40The item's first version and each subsequent version.
Retained item versions per project25,000Item versions across all datasets in the project.
Payload per item406,666 bytesThe combined size of input, expected_output, source_output, and metadata as compact UTF-8 JSON.
JSONL export250 MBThe entire file. A dataset with large items can exceed this limit even when each item fits within its payload limit.

Editing, archiving, or restoring an item can add a version. Each version counts toward both version limits until PostHog deletes it under the retention policy.

Report results with offline evals

After your CI run scores each dataset item, send the results to PostHog with offline evals. Each run appears as an experiment, with one row per test case and one column per evaluator, so you can compare runs and find the items that regressed.

Use a dataset in offline evals

Use a PostHog dataset to manage cases in PostHog or save them from production traces. This example loads a dataset revision before starting the experiment.

This example builds on the offline evals quickstart. Complete the quickstart first. To try it with the demo, create dataset items with string inputs and expected outputs containing tool and arguments, like the cases list in step 3 of the quickstart. Set POSTHOG_DATASET_ID to your dataset's ID. Then, in the step 3 code, replace the experiment creation and the loop over cases with the code below. Keep the same client, scorers, agent, evaluator, error handling, and final complete() call.

snapshot = evaluations.datasets.snapshot(os.environ["POSTHOG_DATASET_ID"])
items = list(snapshot.iter_items())
experiment = evaluations.create_experiment(
name="Support agent from a dataset",
suite_key="support-agent",
dataset_revision_id=snapshot.revision_id,
run_source="local",
)
for row in items:
output = run_agent(row.input)
item = EvaluationItem.from_dataset_item(row, output=output)
for key, result in evaluate_answer(output, row.expected_output).items():
experiment.upload_result(
item=item,
scorer_version_id=scorers[key].current_version_id,
**result,
)

from_dataset_item() / fromDatasetItem() copies the test case and its version into the experiment item. Always supply the new run's output. The dataset's source_output is an earlier answer.

By default, snapshot() selects the latest revision when you call it. To repeat an earlier test set, pass revision=12 in Python or { revision: 12 } as the second argument in TypeScript. The revision must still be available under the dataset retention policy.

Further reading

  • Evaluations overview – online evaluations that score production generations automatically
  • Trace reviews – the manual review workflow for finding failure modes worth adding to a dataset
  • Traces – where dataset items usually come from

Still have questions?

Was this page useful?