Back to blogApprofondimenti

    The quality of generative output: acceptance criteria, reference sets and measurement in operation

    Good is not a measurement. How to write an acceptance criterion for a generative system, which tasks can be checked automatically and what has to be repeated after release.

    ZeroFive.AI November 14, 2026 8 min

    In short. The quality of a generative system cannot be measured until someone has defined it with a written acceptance criterion that a person outside the project could apply. Some tasks, such as code or data extraction, can be checked automatically, and it pays to start from those. The others need a set of real cases signed off by people who know the work, a scoring rubric and a measurement repeated after release. Without declared thresholds a pilot cannot fail, and so it authorises nothing.

    When a generative AI project reaches the meeting where its move to production is decided, sooner or later someone asks whether the system works well, and the answer that comes back is almost always an adjective. The answers are good, the colleagues who tried it are satisfied, the demo convinced the committee. None of these judgements is false, yet none of them is a measurement, and an investment decision taken on an adjective can only be repeated or withdrawn, with no way of knowing what has changed in the meantime.

    It is a problem we are often asked about, including by people who know the technology well: how do you establish that the output of a model, or the work of an agent, is of sufficient quality. The question looks technical and in fact concerns the people who decide, because that definition drives the choice of model, the running cost, the level of human oversight and, for systems that fall under the AI Act, part of the mandatory documentation.

    From criterion to measurement: acceptance criterion, reference set, evaluation and measurement in operation

    Quality as an acceptance criterion

    In traditional software, quality has a familiar shape. A requirement says what the system must do, a test checks that it does it, and the result is always the same for the same input. With generative models this chain breaks at two points: the same input can produce different outputs, and for many tasks there is no single correct answer to compare them with. We wrote about the effect this has on architecture in the piece on probabilistic and deterministic.

    The practical consequence is that quality has to be defined before it is measured, and defining it means writing an acceptance criterion that a person outside the project could apply in the same way. For a summary of meeting minutes it may be the presence of every decision taken and of who owns it, with no invented attributions. For a reply to a customer it may be the correct contractual reference and the expected tone. As long as the criterion stays in the head of whoever built the system, every evaluation is an opinion.

    A written criterion has another, less visible effect. It forces the process owner to declare what margin of error is acceptable, and that number is a business choice no supplier can make on their behalf. An assistant that gets two summaries in a hundred wrong is acceptable for a working group's internal notes and is not for a communication to a supervisory authority.

    Tasks with an objective check

    There is a family of tasks where quality can be checked without human judgement, and it is the one to start from. Code is the clearest case: it compiles or it does not, it passes the tests or it does not, it respects the security rules or it is flagged by the analysis tools. The same holds for extracting data from documents when a reference value exists, for classification into closed categories, for format conversions.

    In these cases the measurement is a pass rate on a set of tests, and it has every property a decision needs. It can be compared between two models, repeated after every update and set against cost. This is why legacy code rewriting programmes are today among the first projects in which a company can say, with a number, whether a smaller and cheaper model produces the same result as a frontier one.

    Anyone looking at a portfolio of use cases should take this into account when choosing the order. For the same expected value, the case where quality can be verified automatically lets you learn faster and defend the next decision better, as we describe in the article on prioritising AI use cases.

    The reference set for tasks with no single answer

    Everything else needs a set of real cases with the expected answer, what practitioners call a golden set. It does not have to be large. Thirty or fifty cases taken from the actual workflow, chosen to cover both the frequent situations and the awkward ones, are enough to bring out most of the defects. We covered how to build it in choosing the model for the task, and the limits of public leaderboards in benchmarks and model evaluation.

    Alongside the cases you need a scoring rubric, that is, the list of criteria against which every answer is judged and the scale used for each. The rubric can be applied by a domain expert or by a second model instructed to do so, and the two routes are not mutually exclusive. The automated judge is cheap and scales, while human judgement is needed to calibrate it and to check on a sample that it is not rewarding answers that are well written and wrong.

    This is where we see the most frequent mistake. The reference set is built by the supplier or the technical team, with cases the system already handles, and the process owners see it only once the result is in. A useful set is signed off by people who know the work, and it also contains the cases where the right answer is to decline to answer.

    Which framework does your company actually need?

    AI Rating measures maturity across the four areas of the model and shows where to start, with priorities and estimated effort.

    Start your AI Rating

    Measurement in operation

    An evaluation carried out before release describes the system at that moment, with that model and those documents. In operation all three change: the provider updates the model, users ask questions nobody had anticipated, the document base grows and ages.

    That is why the measurement has to be repeated, and it is worth deciding from the outset how often and on which signals. The most useful are the ones the system produces by itself during use, such as the share of answers corrected by hand, the requests rephrased several times, the cases handed over to a person. Recording the model used, the response time and the consumption for every request then makes it possible to link quality to cost, which is the relationship on which you decide whether a cheaper model can replace a more expensive one.

    Many platforms offer this observability out of the box, and the risk is to mistake it for a quality measurement. Logs say what happened and what it cost. Whether the result was good is something only a comparison with a criterion can say, and someone has to have designed that comparison.

    The references in the rules

    For high-risk systems the AI Act requires the levels of accuracy and the metrics used to measure them to be declared in the instructions for use, as set out in Article 15 of Regulation (EU) 2024/1689. ISO/IEC 42001 devotes clause 9 to evaluating the performance of the management system, with planned monitoring and measurement. In both cases the premise is the same: the organisation can say what it measures, by what method and who answers for it.

    Outside the high-risk perimeter too, that declaration is what allows a board to approve a release knowing what it is approving. In Italy, since Legislative Decree 160/2026, it is also material that can be requested in court when a system has caused harm.

    Where quality sits in the decision path

    In our method the definition of quality comes in at two moments. The first is the Value Case, where the expected benefit is calculated against a baseline and a non-AI alternative: if you cannot say what level of quality makes the output usable, the benefit is an estimate with nothing underneath it. We discuss it in the article on the value case for an AI project.

    The second moment is the validation gate, where you check that the pilot has reached the declared thresholds before asking for authorisation to release. A pilot with no written thresholds cannot fail, and a pilot that cannot fail authorises nothing. In the AI Rating this capability weighs on the Delivery dimension, while the trust that users and leadership place in the system, which depends on measurement, is read in the Confidence dimension.

    A company that wants to know where it stands can run a simple test on the generative system most used internally, asking its owner which acceptance criterion was written, on how many cases it was verified and when the last measurement was repeated. The answers show whether quality is governed or merely perceived, and it is the point we start from when we begin an AI Rating or in a first assessment meeting.

    Written with the support of Claude, reviewed and approved by Fabio Lalli on November 11, 2026. How we use AI

    Frequently asked questions

    How do you measure the quality of generative output?

    By first writing an acceptance criterion and then verifying it on a set of real cases with the expected answer. The judgement can come from a domain expert, from a second model instructed with a rubric, or from both.

    What is a golden set?

    A set of cases taken from the real workflow, each with the expected answer. Thirty or fifty cases, chosen to cover frequent and awkward situations, are enough to bring out most of the defects.

    Which tasks can be checked without human judgement?

    Those with an objective outcome: code that compiles and passes the tests, data extraction with a reference value, classification into closed categories, format conversions.

    Are platform logs enough to measure quality?

    No. Logs say what happened, with which model and at what cost. Whether the result was good is established only by comparison with a criterion defined by the company.

    Does the AI Act require accuracy to be declared?

    For high-risk systems, yes: Article 15 requires the levels of accuracy and the relevant metrics to be declared in the instructions for use.

    Want to discuss this for your company?

    30 minutes with us to figure out where to start, or an AI Rating to measure your starting point.

    #output quality#evaluation#golden set#observability#value case#AI Rating
    Share

    Keep reading