This post covers how BEval Studio builds evaluation datasets: pulling test cases from production logs, uploading external test sets, what a dataset contains, and how eval runs work against your connected models.
Where test cases come from
Every team evaluating LLMs hits the same question: where do the test cases come from?
- Hand-written cases are biased toward scenarios you already thought of. They miss the edge cases, malformed inputs and queries that mix three document types you never expected to see together.
- Synthetic data means testing your model on inputs another model generated. Its failure modes rarely overlap with those of real traffic.
The best eval data comes from production: the queries your system actually handles, the outputs it actually produces, the failures users actually hit. Getting that data into an eval pipeline is usually manual work. Export logs, clean them, format them, upload them somewhere, hope the schema matches.
In BEval Studio it takes two clicks.
Curating datasets from production logs
Every log in BEval Studio can become a data point. There are three ways to add them.
From the log list
Filter logs by kind, status, score range, date or review status, then check the ones worth testing against. A floating action bar appears: 3 selected - Add to Dataset - Clear. Click Add to Dataset, pick or create a dataset, and you're done.
From a single log
Reviewing a log that would make a good test case? Maybe it's a tricky edge case the model handled well, or a failure you want to regression-test. Hit Add to Dataset in the top-right corner.
From the dataset page
Open a dataset and click Add from Logs. A log browser opens in a drawer, with the same filters and multi-select as the main log list. Confirm, and the logs go straight into the current dataset.
Provenance
Every point added this way gets a FROM LOG badge in the dataset view, linking back to the original log. You can always trace a test case to the exact production event.
Uploading external test sets
Some eval data doesn't come from production. It comes from manual annotation, research collaborations, or domain experts who keep curated test sets in spreadsheets. BEval Studio accepts JSON, JSONL and CSV.
Uploads are grouped into batches, named containers for related files. Each file in a batch has its own status: pending, processing, done or failed. If a file fails (bad format, schema mismatch), you retry that file without re-uploading the whole batch.
During upload, you map your file's fields to BEval's schema:
| Your column or key | Maps to |
|---|---|
| The prompt or source text | input |
| Ground truth, if you have it | expected output |
| Everything else (tags, version labels, annotator notes) | metadata, automatically |
Uploaded points carry an UPLOAD badge with the batch name and source file, so their origin is as traceable as log-sourced points.
What a dataset contains
A dataset is a named container of data points. Each point has:
| Field | What it holds |
|---|---|
| Input | The prompt, query or source document, stored as structured JSON. Supports multimodal inputs via attachments. |
| Expected output | Ground truth, if available. It's optional; you can run evals without it. |
| Metadata | Free-form key-value pairs, shown as collapsible chips instead of raw JSON. |
| Provenance | Where the point came from and when it was added. |
| Attachments | Images, PDFs or audio files for multimodal inputs. |
The dataset detail page has three tabs:
- Points: every data point with its provenance badge, input preview, expected output indicator, metadata chips and attachment count. Filter by source type and search across metadata.
- Uploads: the upload history. Batches, the files in each, processing status, row counts and errors. Failed files have a retry button.
- Runs: every eval run against this dataset, with the model, rubric, score and run time.
Running evals
Once you have a dataset, you can run it against any model connected through BEval Studio's model integrations:
- Click Run Eval on the dataset.
- Pick a connected model.
- Optionally attach a rubric for structured scoring.
- Confirm.
The run dispatches asynchronously. Each point's input goes to the selected model, the output is captured, and if a rubric is attached, judges score the result. Results arrive per data point, so you don't wait for the whole run to finish.
The eval run detail page shows:
- overall progress (completed points out of total)
- aggregate scores
- per-point results: input, model output, score bar, latency and judge dimension breakdowns
- failures, with error messages
You're testing against the queries your system handles in production, curated by people who know your domain, and scored with your own rubrics and judge configurations. Not against synthetic benchmarks.
The regression loop
Put together, the workflow looks like this:
- Your system runs in production. BEval captures and evaluates every log.
- You spot a failure class, such as hallucination on one document type or missing fields on a new input format.
- You pick those failing logs into an eval dataset.
- You fix the prompt, swap the model or adjust the pipeline.
- You run the dataset against the new configuration.
- You see, per data point, whether the fix worked on the exact production cases that failed before.
That's regression testing for LLMs, with provenance and scoring at every step.
If you want to know whether your LLM changes actually improved things, book a call. We'll show you the dataset-to-eval pipeline on your production data.