Skip to content

Free browser tools that respect your files: everything runs locally, nothing is uploaded.

Prompt A/B Tester

Variants × dataset × assertions — pass rates, an optional LLM judge, and a hard cost-stop guard.

Variants (2)

Dataset (3 cases)

Assertions

Run controls

2 × 3 = 6 model calls · trial covers 3

Report

The grid appears here: one row per dataset case, one column per variant, pass/fail per cell, aggregate pass rates, and the winner. Run a test to fill it.

Quick Answer

To A/B test prompts, add two or more variants, paste or upload a dataset of test inputs (CSV, TSV, or one per line), optionally define assertions — contains, regex, JSON-valid, max length — and press Run test. Every variant runs against every case (two at a time, with a live cost meter and a hard "stop at 0.50); the report table shows pass/fail per cell, aggregate pass rates per variant, and the winner. An optional LLM-as-judge scores each output against your rubric — labeled model opinion, never a benchmark. The previous run of the same dataset is kept for regression comparison. Free trial covers 3 model calls a day; full runs use your own API key.

How It Works

1

Add 2–6 prompt variants — {{input}} in a variant is replaced with each dataset row

2

Paste or upload the dataset (one case per line, optional expected value after "=>"; up to 40 cases) and set assertions: contains, regex, valid JSON, max length

3

Optionally enable the LLM judge with your rubric (it doubles the model calls — every score is labeled model opinion)

4

Press Run test: cells fill pass/fail two at a time under a live cost meter and the stop-at guard; the winner column is highlighted and the previous run of the dataset is kept for regression comparison

Key Facts

  • 2–6 prompt variants tested against a dataset of test inputs (paste CSV/TSV, upload a file, or reuse a saved dataset)
  • Assertions evaluated deterministically in your browser: contains, regex, valid JSON, max length
  • Optional LLM-as-judge with your own rubric — scores labeled as model opinion
  • Live cost meter with a hard stop guard ("Stop at 0.50) so a big run can never surprise you
  • Aggregate pass rates per variant, winner highlight, and export of the report as CSV or JSON
  • Regression compare: the previous run of the same dataset is kept, and pass-rate deltas are shown
  • Runs are sequential, two at a time — datasets stay polite to your provider

Frequently Asked Questions

How are assertions evaluated?

Deterministically, in your browser: "contains" checks a substring, "regex" matches a pattern, "valid JSON" parses the output, "max length" caps characters. An output passes only if every assertion passes. The LLM judge is separate and opt-in — its scores are opinions against your rubric, clearly labeled.

What does the free trial cover here?

The trial budget counts model calls, not button clicks: 3 calls a day is enough to smoke-test one variant on three cases. Full dataset runs use your own API key — the cost meter and the stop guard show exactly what a run costs as it happens.

How does regression comparison work?

Each finished run is kept locally per dataset (the latest previous run). Running the same dataset again shows pass-rate deltas per variant — green where the new prompt improved, red where it regressed.

Is this a replacement for promptfoo or PromptLayer evals?

It covers the core of what they charge for — variants × datasets × assertions × judge, with reports — as a free browser tool on your own key. It does not run from CI or store shared team results; for CI pipelines, the open-source promptfoo CLI is the right complement.