Observability that goes beyond logging

This post walks through the BEval Studio dashboard: how logs get in, what each row of the home view tells you, and how the log list and review workflow fit together.

Logging is not observability

Most teams shipping LLMs already log. Inputs and outputs land in a database, a file or a third-party tool, and you can look up any single request.

That answers "what did the model say?" It doesn't answer the questions that matter in production:

• Is quality getting better or worse over time?
• Which failure types are most common right now?
• Did last Tuesday's prompt change improve anything?
• Which outputs need human review?
• What does this cost per request, per day, per model?

The dashboard answers these directly. You don't write queries or build your own analytics.

Unified log ingestion

Everything starts with one API call. Your inference pipeline sends a log entry with whatever fields it has:

{
  "name": "invoice-extraction",
  "kind": "vlm",
  "input": "Patient: John Doe, DOB: 1985-03-12...",
  "output": "{ \"patient_name\": \"John Doe\", ... }",
  "extracted_json": { ... },
  "model_id": "claude-sonnet-4-6",
  "latency_ms": 843,
  "tokens_in": 1200,
  "tokens_out": 340,
  "cost_usd": 0.0024
}

Core identity and I/O fields are required. Everything else is optional.

The same table handles a chat completion, a VLM document extraction, an agent trace, an embedding call or an OCR pipeline. There are no separate log types to configure. You send what you have and BEval structures it.

Deterministic evaluation runs on every entry at ingest. Selected entries are queued for LLM judge evaluation in the background, so the scores are already there when you open the dashboard.

The home dashboard

The home page is driven by a timeframe: today, 7 days, 30 days or 90 days. Change it and every panel updates.

Row 1: Vital signs

Four stat cards: Total Logs, Average Latency, Success Rate and Pending Reviews. If success rate drops or reviews pile up, you see it here first.

Row 2: Evaluation scores

Three cards: Average Deterministic Score, Average Judge Score and Rubric Pass Rate (for your default rubric).

Read them together. A high deterministic score with a falling judge score means a semantic problem your structural checks aren't catching. A falling pass rate on one rubric tells you which criteria are slipping.

Row 3: Trends over time

A line chart of det_score, judge_score and latency. This is where regressions show up.

A prompt change on Monday appears as a score shift on Tuesday. A new document type in your corpus appears as a latency spike. This view turns "something feels off" into "quality dropped 4% starting March 12th."

Row 4: Rubric breakdown

One bar per rubric criterion, showing the average score across the timeframe. If four criteria are green and one (say, line item completeness) is red, you know where to look.

Row 5: Judge dimensions

Horizontal bars for hallucination, completeness and semantic accuracy, or whatever dimensions your judge configuration evaluates. They tell you what kind of quality problem you have, not just that you have one.

Row 6: Failure analysis

A table of deterministic check failures ranked by count and share of total: missing required fields, type mismatches, range violations, empty arrays. Use it to decide what to fix first.

The log list

When the dashboard says something is wrong, the log list is where you find out why.

Filters:

• Kind: LLM, VLM, agent, embedding
• Status: success, failure
• Score range
• Review status: unreviewed, reviewed, flagged
• Date range

Logs sort lowest score first by default, so the worst outputs are at the top.

Each row shows the external ID, a kind badge, det score, judge score, latency, review status and timestamp. Opening a log gives you the input and output text, extracted JSON (editable for correction), thinking trace, judge scores with reasoning, deterministic check results and a review notes panel.

Human review workflow

Some outputs need a domain expert to confirm whether the model got it right. Automated checks can't cover everything.

Every log carries a review status: unreviewed, reviewed or flagged. Pending reviews show up as a top-level metric, and the log list filters down to unreviewed or flagged entries.

A reviewer opening a log sees the input, output, extracted JSON, thinking trace and all automated scores. From there they can:

• Edit the extracted JSON to fix errors. The corrected version is stored separately as corrected_json.
• Add notes on what was wrong or right.
• Submit the review, which updates the status and records who reviewed it and when.

Over time this builds a set of human-validated corrections you can use for prompt changes, fine-tuning datasets or rubric adjustments.

Where to look for what

Every number comes from automated evaluation on your production traffic, not a benchmark or a manual spot-check.

If your monitoring today is "check the logs when someone complains," book a call and we'll show you the dashboard.