Quick Answer
To A/B test prompts, add two or more variants, paste or upload a dataset of test inputs (CSV, TSV, or one per line), optionally define assertions — contains, regex, JSON-valid, max length — and press Run test. Every variant runs against every case (two at a time, with a live cost meter and a hard "stop at 0.50); the report table shows pass/fail per cell, aggregate pass rates per variant, and the winner. An optional LLM-as-judge scores each output against your rubric — labeled model opinion, never a benchmark. The previous run of the same dataset is kept for regression comparison. Free trial covers 3 model calls a day; full runs use your own API key.
How It Works
Add 2–6 prompt variants — {{input}} in a variant is replaced with each dataset row
Paste or upload the dataset (one case per line, optional expected value after "=>"; up to 40 cases) and set assertions: contains, regex, valid JSON, max length
Optionally enable the LLM judge with your rubric (it doubles the model calls — every score is labeled model opinion)
Press Run test: cells fill pass/fail two at a time under a live cost meter and the stop-at guard; the winner column is highlighted and the previous run of the dataset is kept for regression comparison
Key Facts
- 2–6 prompt variants tested against a dataset of test inputs (paste CSV/TSV, upload a file, or reuse a saved dataset)
- Assertions evaluated deterministically in your browser: contains, regex, valid JSON, max length
- Optional LLM-as-judge with your own rubric — scores labeled as model opinion
- Live cost meter with a hard stop guard ("Stop at 0.50) so a big run can never surprise you
- Aggregate pass rates per variant, winner highlight, and export of the report as CSV or JSON
- Regression compare: the previous run of the same dataset is kept, and pass-rate deltas are shown
- Runs are sequential, two at a time — datasets stay polite to your provider
Frequently Asked Questions
How are assertions evaluated?
Deterministically, in your browser: "contains" checks a substring, "regex" matches a pattern, "valid JSON" parses the output, "max length" caps characters. An output passes only if every assertion passes. The LLM judge is separate and opt-in — its scores are opinions against your rubric, clearly labeled.
What does the free trial cover here?
The trial budget counts model calls, not button clicks: 3 calls a day is enough to smoke-test one variant on three cases. Full dataset runs use your own API key — the cost meter and the stop guard show exactly what a run costs as it happens.
How does regression comparison work?
Each finished run is kept locally per dataset (the latest previous run). Running the same dataset again shows pass-rate deltas per variant — green where the new prompt improved, red where it regressed.
Is this a replacement for promptfoo or PromptLayer evals?
It covers the core of what they charge for — variants × datasets × assertions × judge, with reports — as a free browser tool on your own key. It does not run from CI or store shared team results; for CI pipelines, the open-source promptfoo CLI is the right complement.