AI Testing & Evaluation
Know whether your AI feature works — and whether the next change makes it better or worse.
Evaluation datasets from real cases, automated scoring, LLM-as-judge calibrated by people, regression tests in CI and monitoring in production.
AI TESTING & EVALUATION
Stop judging AI quality by trying a few prompts.
Turn "it seems fine" into numbers you can track, compare and defend.
Most AI features are shipped after someone on the team tried a dozen questions and liked the answers. Then the prompt is tweaked, the model provider releases a new version, the documents in the knowledge base change — and nobody can say whether the feature got better or quietly got worse. Users notice before the team does.
We build the evaluation layer that makes AI behave like the rest of your software: a test suite built from real cases, automated scoring for the criteria that matter to your business, and checks that run on every change and keep running in production. It is the same discipline our team has applied to production software since 2013, adapted to systems whose output is not deterministic. It works for chatbots, RAG knowledge assistants, AI agents, extraction pipelines and classic machine learning models.
Evaluation datasets
We collect real questions, documents and edge cases from your logs, support tickets and domain experts, label the expected outcome and keep the set versioned. Synthetic cases fill the gaps, but real ones come first.
Automated evals
Each answer is scored on accuracy, groundedness in the sources, output format, safety and tone. Deterministic checks where possible, an LLM judge where needed, calibrated against human graders before anyone relies on it.
Regression and monitoring
The suite runs in CI on every prompt, model or retrieval change and blocks the merge when quality drops. In production we track failure rates, drift and cost, and feed new failures back into the dataset.- Versioned evaluation dataset built from real cases
- Scoring rubric agreed with your domain experts
- Automated checks for accuracy, groundedness, format, safety and tone
- LLM-as-judge calibrated against human graders
- Regression suite for prompt, model and retrieval changes
- Side-by-side model comparison: quality, cost and latency
- Load and cost testing before launch
- Eval suite running in your CI on every change
- Production monitoring for drift, failures and spend
- Dashboards and reports your team and auditors can read
WHAT WE MEASURE
Quality, cost and speed — on your data, not on public benchmarks.
Public leaderboards tell you how a model does on someone else's questions. What you need to know is how it does on your contracts, your product catalogue, your support tickets and your users' wording. So every comparison we run uses your evaluation set: the current model against a cheaper one, a hosted model against an open-weight model on our own GPU cloud, the old prompt against the new one. You get a table with quality scores per criterion, cost per thousand requests and latency at the percentiles that matter, and you choose the trade-off with the facts in front of you.
Before launch we also run load and cost tests: realistic traffic patterns, long and short inputs, retries and rate limits, so you know what a busy day will cost and where the system slows down. Safety checks here cover the expected behaviour — refusing out-of-scope requests, not leaking data, keeping the right tone. Deliberate attacks such as prompt injection and jailbreaks are the subject of AI security and red teaming, and the two suites share the same harness.
Tools we work with include LangGraph, OpenAI, Anthropic Claude, open-weight models such as Llama, Mistral and Qwen served with vLLM, PostgreSQL with pgvector, and the CI system you already use. Results, datasets and logs can stay on infrastructure we operate in Cluj-Napoca on our own network (AS56951) when the data cannot leave your control.
HOW WE WORK
From "seems fine" to a quality bar you can enforce
Small first, then automated, then continuous.
Define what good means
Together with the people who own the process, we write down what a correct, useful and acceptable answer looks like, and which mistakes are expensive.
Build the dataset
We gather real cases, label expected outcomes with your experts and cover the known failure modes: ambiguous questions, missing information, out-of-scope requests.
Measure the baseline
We wire up the scorers, calibrate the judge against human grades and run the current system. Now there is a number, and a list of the cases it gets wrong.
Compare and improve
Prompts, retrieval settings and models are compared on quality, cost and latency. Changes are kept only when the numbers say they help.
Gate every change in CI
The eval suite runs on every pull request, with thresholds per criterion. A regression is visible in the review, not in a customer complaint.
Monitor in production
We sample live traffic, score it, alert on drift and rising failure rates, and turn new failures into test cases, so the suite keeps up with reality.RISKS WE HANDLE
Where AI evaluation usually goes wrong
Testing on the wrong data. A set written by the developers who built the prompt tends to pass. We draw cases from real usage and keep a held-out part that nobody tunes against.
Trusting an uncalibrated judge. An LLM judge can prefer longer answers or answers in its own style. We measure its agreement with human graders and re-check it when the judge model changes.
One average hiding the problem. An overall score can rise while a critical category gets worse. Results are broken down by criterion and by segment, with stricter thresholds where mistakes cost more.
Silent changes upstream. Hosted models are updated, documents are edited, user behaviour shifts. Scheduled eval runs and production sampling catch the change even when your code did not move. Documented tests and monitoring also feed directly into EU AI Act compliance work.
FAQ
Questions teams ask before they start
We already test our app. Why do AI features need a separate evaluation suite?
Classic tests check that the same input gives the same output. A language model can give a different answer every run, and an answer can be well formed and still wrong. Evals score the quality of answers across many realistic cases and tell you whether a change made things better or worse on average and on the cases that matter most.
Can we trust an LLM to grade another LLM?
Only after it has been checked against people. We have domain experts grade a sample of answers, compare their scores with the judge model, and adjust the rubric until they agree well enough for the decision you need to make. Criteria that can be checked with code, such as format, citations or forbidden content, are checked with code, not with a judge.
How many test cases do we need to start?
Fewer than most teams expect. A small, carefully chosen set that covers your main use cases and known failure modes is already useful for catching regressions. The set then grows from real traffic: every bug report and every bad answer found in production becomes a new case.
Which models and frameworks do you work with?
Hosted models such as OpenAI and Anthropic Claude, and open-weight models such as Llama, Mistral or Qwen served with vLLM, including on our own GPU cloud. The eval harness is model-agnostic, so the same suite can compare all of them on your data. We can use open-source eval tooling or a lightweight harness in your own codebase, depending on what your team will maintain.
Does evaluation help with the EU AI Act?
It produces much of the evidence you will need: documented test data, accuracy and robustness measurements, records of changes and how they were validated, and logs from production monitoring. It does not replace a legal assessment of your system's risk category.
Have an AI feature you cannot measure yet?
Tell us what it does and how you check it today. We will propose a first evaluation set and tell you honestly what it can and cannot prove.