No release ships until model evaluation and assurance say it may.

Answers can get worse while the code stands still.

When conventional software fails, it usually throws an error. A system built on a model can get worse without one: extraction slips on one supplier's layout, answers turn confident where they should refuse, and a summary starts to drop the one clause that mattered. None of it shows on a dashboard of uptime and response time.

Part of the change is not yours to schedule. Providers such as Anthropic and OpenAI retire models on published timetables, with notice ranging from months for most models down to weeks for previews, and each replacement behaves differently. The notice periods are under model operations.

Engineering already has the remedy: a representative test set, run on every change, behind a gate that refuses the release when the numbers move the wrong way. Here the set has to come from your own work, and the thresholds are agreed with the people who carry the risk.

Six parts, from test set to audit file.

• An evaluation set from your work. Real cases with agreed answers, the hard, ambiguous and refusal cases included.
• Refusal and safety cases. Explicit tests that the system declines what it should, and says 'I do not know' when the answer is missing.
• A release gate. The suite runs on every change to the prompt, the model, the retrieval or the data, and a regression stops the release before it ships.
• A model-upgrade test. The same suite run against a candidate model, so switching provider or version is decided on numbers.
• Drift monitoring in production. Sampled live traffic scored against the same criteria, with an alarm when the distribution moves.
• Evidence you can hand an auditor. What was tested, when, on which version and with what result, filed, versioned and reproducible.

Five kinds of case, written with your people.

• Must never be wrong. Cases where one error costs real money or trust. The tolerance is zero, and any failure blocks the release.
• The everyday bulk. Enough ordinary cases to show the common path still works, scored against an agreed rate.
• The hard edges. Bad scans, pages that mix two languages, the supplier whose invoices break every template, all chosen on purpose.
• The ambiguous. Cases where the business itself has no single right answer, so the test is whether the system escalates.
• The hostile. Inputs drawn from the OWASP Top 10 for LLM Applications (2025) that apply to your system: an instruction hidden inside a document (LLM01, prompt injection), a figure shown to a role that may not see it (LLM02, sensitive information disclosure), a tool called that should not have been (LLM06, excessive agency), and a confident answer with nothing behind it (LLM09, misinformation).

The hostile cases are regression tests against known attack patterns, while red teaming is a separate discipline that we do not sell. Below, a worked example.

A release ships only when the gate passes.

Two rules keep the gate honest.

First, every new behaviour ships with a test that proves it, and we watch that test fail on the real defect before we trust it.

Second, an evaluation that cannot reach the model counts as a failure, so the gate reads red.

A refused release leaves the live version serving. Our own platform ships through the same release gate, and its record for the 10 September 2026 release shows no check failing and no client document missing a value.

Each artefact maps to NIST's risk framework.

The NIST AI Risk Management Framework, AI RMF 1.0 (NIST AI 100-1, released 26 January 2023), is a voluntary framework built on four functions: Govern, Map, Measure and Manage. Its Generative AI Profile (NIST AI 600-1, July 2024) adds risks specific to generative systems, among them confabulation, the confident production of false content. NIST states that AI RMF 1.0 is being revised.

Your risk or compliance team can then file each piece of evidence where it belongs, subcategory by subcategory.

Your auditor decides what the evidence proves.

SolaryxDev holds no AI management system certification, ISO/IEC 42001 included, and certifies no one else's system. We make your auditor's check easy. Each result traces to a version and can be run again, and every case is written for a reader who was not in the room.

Five steps, from the first case to production.

• Find the cases that matter. With the people who do the work, we separate the errors that must never happen from the merely annoying ones and from the questions nobody can agree on.
• Write them down as tests. Input, expected behaviour and tolerance for each, so ambiguity is settled on paper before it causes an incident.
• Baseline what you have. Whatever runs today is scored before anything changes.
• Wire the gate into the release path. A regression cannot ship, including one caused by a provider change you did not ask for.
• Watch production. Sampling, scoring and alerting, with thresholds reviewed as the system earns them.

The suite stays with you when we finish, whoever maintains the system afterwards.