AI-Assisted NDT: A Practical Validation Guide for Buyers

Define the AI task before discussing its accuracy

AI-assisted NDT can mean several different things: highlighting regions for a reviewer, sorting an examination queue, classifying an indication, measuring a feature or drafting report text. Each changes a different part of the inspection workflow. A useful evaluation begins by stating the exact proposed task, the input, the output, the intended user and the decision the output may influence. A single claim about an AI platform cannot stand in for that definition.

For a U.S. inspection business or asset owner, the purchasing question is whether a defined system can support a defined use under the organization's actual conditions. This guide provides an original evaluation workflow and a hypothetical arithmetic example. It does not establish NDT acceptance criteria, a universal test-set size or permission to automate a final acceptance decision. Those decisions require the applicable technical, quality and customer review.

Keep a general AI framework separate from an NDT authorization

NIST describes its AI Risk Management Framework as a voluntary resource for considering trustworthiness in the design, development, use and evaluation of AI systems. The framework is useful context for organizing responsibilities and evidence. It is not an NDT code, a product certification or a replacement for an employer's and customer's examination requirements.

Write down what the proposed AI output is allowed to do. A system that suggests where a qualified reviewer should look is different from one that removes examinations from review. A system that drafts a sentence is different from one that decides whether a component meets requirements. If a pilot later changes that authority, treat it as a change of intended use and review the evidence accordingly.

Identify the people responsible for the examination process, software configuration, evaluation dataset, customer data and final decision. Do not assume the software supplier owns all these responsibilities. The inspection organization still needs someone who can explain the complete workflow, including what happens when the software is unavailable, rejects an input or produces an output that conflicts with other evidence.

Define the unit of evaluation

Before counting errors, decide what one evaluated item represents. It might be an image, a component, an examination volume, an indication or a report. These units are not interchangeable. A model that identifies one indication in a component containing several relevant indications can appear successful at component level while missing findings that matter to the actual task. The evaluation record should preserve the distinction.

Define how repeated flags, overlapping regions and incomplete inputs are counted. If the software identifies the same feature several times, the review burden differs from one correct flag. If the system declines to process difficult data, those cases should remain visible rather than disappearing from the denominator. Agree the counting rules before seeing the final results so they cannot be adjusted to make one product appear stronger.

State what counts as a correct reference outcome and how uncertainty will be handled. A historical report may be useful evidence without being an unquestionable label for every region in the original dataset. Record the reference process, reviewer qualifications, unresolved disagreements and any supporting examinations. A well-managed uncertainty category is more informative than forcing every ambiguous case into a confident positive or negative label.

Separate training, tuning and independent evaluation data

Ask how the supplier separated the data used to develop the system from the data used to evaluate it. The separation must address related observations, not only different file names. Cropped images, repeated acquisitions and neighboring views of the same component can carry closely related information. A test that contains those relationships may answer a narrower question than performance on genuinely new production work.

NIST's AI Technology Evaluation program describes blind testing in a sequestered environment intended to reduce contamination between training and testing. The cited program is not an NDT approval scheme; it illustrates why independent evaluation information matters. For a purchasing trial, agree which cases remain unavailable for model development and who controls access to them.

Maintain a dataset manifest with source identifiers and the rules used to group related records. Record which items were used for development, configuration selection and the final evaluation. If a supplier studies a failed test case and changes the model, that case has now influenced development. Retain the original result and plan an appropriate further independent evaluation rather than presenting a rerun on familiar cases as an untouched test.

Represent the work population and the difficult boundaries

Build the evaluation population around the intended application. Consider the material and component families, examination configurations, acquisition systems, sites and data conditions that the deployment is expected to encounter. The relevant dimensions depend on the actual task. Ask the technical team to identify which differences could change the relationship between the input and the required output, rather than collecting variation merely to make a dataset large.

Include the boundaries of the supported scope. The system should have a defined response to unsuitable, incomplete or unsupported input. A confidently displayed result for data outside the evaluated application can be more misleading than a clear refusal to process it. Test whether the user can recognize that boundary and whether the workflow provides a practical route to continue through an authorized alternative.

Document which populations are underrepresented. A strong overall result can coexist with weak or uncertain performance in a small but important subset. Report the relevant counts and limitations rather than presenting every subgroup percentage as equally reliable. Qualified evaluation personnel should determine the necessary study design and uncertainty treatment; there is no universal sample size in this article that establishes suitability for every NDT application.

A worked example: why one percentage is insufficient

Consider a purely hypothetical classification exercise containing 1,000 independent items. The agreed reference identifies 10 positive items and 990 negative items. Suppose a model flags nine of the positive items and also flags 99 negative items. It misses one positive item and correctly leaves 891 negative items unflagged. These numbers are invented solely to explain the arithmetic; they are not results from an NDT product or a recommended validation study.

The model's recall for the positive class is nine divided by ten, or 90 percent. Its precision among flagged items is nine divided by 108, approximately 8.3 percent. Its overall accuracy is 900 correct classifications divided by 1,000, or 90 percent. The reviewer receives 108 flagged items, of which nine correspond to reference positives. Each number describes a different aspect of the same result.

Now consider a system that never flags anything in that example. It would correctly label 990 negative items and report 99 percent overall accuracy, while missing every positive item. That higher accuracy would not make it useful for finding the positives. The example shows why an imbalanced population can make an impressive headline percentage a poor purchasing criterion. It does not establish acceptable error rates for any examination.

Ask for the underlying counts, the evaluation unit and the population description whenever a supplier presents a metric. Also ask whether the result describes the algorithm alone or the complete reviewer-assisted workflow. Neither the arithmetic above nor a general classification score is a probability-of-detection demonstration for a defined NDE system. The appropriate technical evidence must match the claim being made.

Select the operating configuration before the final test

A system may allow thresholds or other settings that change the balance between missed findings and false alarms. Agree how those settings will be selected and document the configuration used for the final evaluation. If the final test itself is repeatedly used to choose the setting, its independence is reduced. Separate configuration development from the evidence used to judge the selected configuration.

Compare suppliers using equivalent tasks and clearly described operating conditions. One demonstration may show a configuration that finds more candidates while another emphasizes a smaller review queue. A comparison is incomplete unless it describes both the error outcomes and their consequences in the intended workflow. The responsible organization should define the decision criteria before results arrive, including which unresolved limitations prevent the proposed use.

Evaluate the person and software working together

The Measure section of NIST's AI RMF Playbook emphasizes selecting measurements that fit the purpose and documenting risks that cannot be measured. It also discusses the importance of context and human interaction. For an NDT buyer, this supports looking beyond an isolated model score to the real decision process, while leaving the actual study design to qualified reviewers.

In an original workflow trial, ask intended users to perform representative tasks with the proposed system and record what they do with its output. Can they open the relevant original data, identify unsupported inputs and explain why they accepted or rejected a suggestion? Does a highlighted region draw attention away from unhighlighted information that remains within their review responsibility? Those observations can expose problems that a batch classification score will not reveal.

Measure time consistently. Include data preparation, failed imports, review of false alarms, escalation, correction and final documentation. A faster model response does not necessarily reduce the full examination or reporting effort. Compare equivalent completed tasks and record changes in error outcomes alongside timing. Do not advertise a percentage productivity gain from a demonstration that omitted the difficult cases or used a different review responsibility.

Keep generative text assistance in its own evaluation

An assistant that drafts a report or procedure performs a different task from an image classifier. NIST's Generative AI Profile is a cross-sector companion to the AI RMF that addresses risks associated with generative systems. It does not validate generated technical content or establish that an NDT procedure complies with a particular code.

For a drafting assistant, define the authorized source material and the content it may produce. Review whether it preserves supplied facts, identifies missing inputs, distinguishes assumptions and avoids inventing technical requirements or results. Include cases where sources conflict, a required input is absent or the requested conclusion is unsupported. A polished document should not hide that the necessary technical decision has never been made.

Keep the draft status visible until the required review and authorization are complete. NDT Connect's procedure generator provides a structured review draft in its default workflow; it does not issue an approved inspection procedure. When evaluating any drafting product, separate document completeness from technical correctness and approval. A longer output or a professional layout does not establish either of the latter.

Agree how the supplier may use the evaluation records

Before sharing customer examinations, identify who has authority to provide them and what the supplier may do with the information. Evaluation access, product support and model training are different purposes. Record the agreed purpose, permitted recipients, retention arrangement and return or deletion process. The buyer should understand whether submitting a difficult case for support also makes it available for future development, rather than discovering that interpretation after the trial.

Prepare the shared package so it contains the information needed for the agreed task. Removing direct identifiers may still leave customer information in image overlays, file names, report text or associated metadata. Have the responsible data owner review the package and its permitted use. Preserve a controlled mapping internally when it is necessary to investigate results, and ensure the technical evaluation remains meaningful after any authorized changes to the shared records.

Control software versions and changes to the use

Record the tested model or software version, relevant preprocessing, configuration and output format. The retained result should be reproducible enough for an authorized reviewer to understand what was evaluated. Ask the supplier how updates are communicated, whether a previous configuration can be identified and what evidence accompanies a material change. A product name alone is not a sufficient description of the tested system.

Changes may affect different parts of the evidence. A user-interface change could alter how reviewers notice a warning, while a model change could alter classification behavior. A new acquisition system may change the inputs even when the model is unchanged. Assign responsibility for reviewing these changes and deciding what further checks or evaluation are required. Avoid both silent updates and a vague promise that every future version inherits the original result.

Define the permitted deployment envelope in the release record. Include the users, tasks, input families, operating configuration, review obligations, limitations and escalation route. If the organization expands the task or population, review that expansion as a new evidence question. This makes the original evaluation useful without allowing a narrow successful pilot to become an unrestricted production claim.

Make failures visible after deployment

Agree how users report a missed finding, a misleading output, an unsupported input or a software interruption. Link the report to the relevant version and source records without exposing customer data beyond authorized recipients. The immediate response should follow the actual quality and technical process, including any necessary review of affected work. The software supplier's support ticket is only one part of that response.

Plan periodic review of actual use, including whether users are applying the system outside its documented scope. Preserve enough information to investigate meaningful changes in the work population and performance. Monitoring should support a defined decision: continue, restrict, investigate, change or stop the affected use. A dashboard that collects counts without an owner for the resulting action provides little assurance that an emerging problem will be addressed.

Build an evidence request before signing the software order

Ask the supplier for the intended-use statement, evaluation protocol, dataset separation approach, underlying outcome counts, relevant subgroup results, known limitations and change-control arrangements. Request a demonstration of the review workflow using appropriately authorized representative data. Keep claims that have not been demonstrated in an open-items register, with an owner and a consequence for the purchasing decision.

Use the remote NDT data review guide to define access and handover questions, and the inspection data manifest guide to organize the evaluation package. If an inspection provider will operate the software for you, include these evidence requirements in the inspection service request. The commercial objective is a supported, bounded capability that the team can use and review, with the remaining uncertainty visible.

Sources and editorial scope

Primary sources checked October 31, 2026: NIST AI RMF overview, NIST AITE overview, NIST AI RMF Playbook: Measure, and NIST Generative AI Profile. These are general AI resources, not NDT product approvals. The arithmetic example and buyer workflow are original editorial explanations, with no claimed product benchmark or production validation result.