| Takeaway | Detail |
|---|---|
| Use a 100-sentence test set. | Build 100 sentences and stratify them by term frequency. |
| Include at least 30 domain-term sentences. | At least 30 of the 100 sentences must contain domain terms. |
| Require 95% terminology consistency. | At least 95% must meet the pre-registered terminology criterion. |
| Measure consistency at the document level with exact matching. | Compute the document-level exact-match rate against a pre-registered termbase, not a token-level average. |
This guide explains how to test terminology consistency with a stratified 100-sentence set, including at least 30 sentences containing domain terms. It defines a 95% threshold using document-level exact matching against a pre-registered termbase.
How terminology consistency is measured
Build the 100-sentence evaluation set before running the system, and register its terminology rules in advance. The set must be stratified rather than composed of convenient examples: at least 30 sentences must contain one or more domain terms, and every term being evaluated must occur at least three times across the set. This minimum gives each term enough observations for a meaningful per-term consistency calculation; a term that appears once or twice cannot support a reliable comparison.
For each term, compare every output occurrence with the preferred form in the pre-registered termbase. A match must be exact, including the term’s approved spelling and capitalization. Variants, synonyms, and inflected forms are errors unless the termbase explicitly lists them as allowable forms. The registration record should therefore include the preferred term, any permitted casing, and every approved variant. A term absent from that record cannot be silently treated as correct, because doing so would change the scoring standard after seeing the results.
Calculate each term’s consistency from its own matching and nonmatching instances. A sentence passes when every domain-term instance in it matches the registered termbase. Report the result at two levels that must agree: a sentence-level rate (passing sentences out of the 100-sentence set) and an instance-level rate (matching term instances out of all expected term instances, e.g. 57 of 60 in the worked example below). The acceptance threshold is applied to the instance-level exact-match rate against the pre-registered termbase, not to a token-level average. A token-level average can conceal a failed sentence by offsetting its errors against many unrelated tokens.
Apply the 95% threshold only after confirming that the scoring denominator is the complete 100-sentence set. Also retain the per-term results as a diagnostic: they reveal which specific terms drive failures and whether any one low-frequency term is distorting the aggregate. Check that all evaluated terms meet the three-occurrence requirement, that at least 30 sentences contain domain terms, and that the termbase was registered before the outputs were inspected. These checks make the result auditable rather than dependent on post hoc judgment.
The reference base should combine established definitions with the project’s preferred usage. Wikipedia’s Glossary of artificial intelligence, Luca Malattia’s Guide to Artificial Intelligence Terminology and Costs, and Scribbr’s AI glossary provide useful starting points for identifying and defining relevant terminology. They do not replace the project termbase: preferred forms, approved exceptions, and evaluation frequencies still need to be fixed in advance.
Evidence from independent sources
Independent terminology sources can serve as external checks on the evaluation set, but they should be treated as evidence for definitions and cost mechanisms rather than as a substitute for a pre-registered termbase. For this section, three sources provide that external grounding: Wikipedia’s Glossary of Artificial Intelligence, Luca Malattia’s Guide to Artificial Intelligence Terminology and Costs, and Scribbr’s Glossary of AI Terms. Their role is to reveal whether the evaluation uses established technical meanings, whether related concepts are clearly separated, and whether likely sources of downstream expense have been considered.
The agent definition used in this guide — an agent that observes a particular environment through sensors and works to achieve a goal for the algorithm — is documented in G2’s glossary of artificial intelligence terms, so attribute it there rather than to Wikipedia. A practical check is to inspect every evaluation sentence containing “intelligent agent” and ask whether the system’s usage preserves both required ideas: environmental observation through sensors and goal-directed action. The term should not drift into a generic reference to an AI application, a chatbot, or a person who makes intelligent decisions. Wikipedia’s glossary is a list of definitions of terms and concepts relevant to AI and its subdisciplines; before relying on it for any specific term’s definition, open the relevant entry and confirm the wording, since the grounding establishes only that the glossary exists and covers AI subdisciplines, not the content of individual entries.
Scribbr’s glossary provides a useful complementary check because it includes entries for both artificial intelligence and AI-detection tools. The editorial rule is simple: preserve the source’s distinction between a system demonstrating machine intelligence and a tool designed to detect AI-generated text. Reviewers should not treat the terms as interchangeable, even when the surrounding prose appears to make them sound similar. This comparison is especially useful for detecting “related-term substitution,” a failure mode in which a plausible phrase replaces the registered term without changing the sentence’s general subject. Each substitution should be reviewed against the registered definition, rather than accepted merely because the replacement is intelligible to a general reader.
Malattia’s guide adds the cost dimension. It distinguishes input tokens from generated output tokens and characterizes output as the higher-cost side of the exchange. For evaluation purposes, that distinction supports a concrete inspection: review corrected outputs for terminology errors and identify which errors require regenerated output rather than an editorial adjustment to existing input. The inference is that inconsistent terminology can create downstream correction work, although the source establishes token pricing mechanics rather than measuring the cost of terminology inconsistency itself. Teams should therefore record both the number of corrected terminology instances and the amount of output that had to be regenerated.
Together, these sources support a documentation practice: cite the definition used for each evaluated term, record any disagreement among glossaries, and document how the chosen definition was registered before evaluation. A reviewer should be able to trace a term to a named source, see why that source’s definition was selected, and distinguish a genuine inconsistency from a legitimate use of a different technical concept. This process makes the evaluation auditable and prevents a familiar-sounding paraphrase from being mistaken for the approved term.
Options for threshold and test set design
Option A is a random sample of 100 sentences from the domain corpus. It is the cheapest design to assemble and preserves the corpus’s natural mix, but it offers no guarantee that rare or specialized terms will appear. If a target term is represented by only a few sentences, a single failure can materially change its estimated consistency, while ordinary sentences add little information about terminology performance. Treat this design as a screening option, not as the basis for a stable benchmark, and report the number of eligible term instances for every evaluated term so readers can distinguish weak estimates from well-supported ones.
Option B is a stratified design: select 100 sentences so that at least 30 contain one or more domain terms, while retaining enough non-term sentences to reflect the corpus. The term-containing subset can be allocated across term-frequency bands, such as rare, moderately common, and common terms, rather than allowing common terms to dominate through their natural prevalence. This balances coverage with statistical power. It also lets evaluators check whether performance differs by frequency band and prevents a strong result on familiar terms from concealing errors on specialized vocabulary.
Option C takes a term-focused approach: construct 100 sentences in which every sentence contains a target term. This maximizes the number of opportunities to observe a terminology decision and may be useful for a diagnostic drill on a small term list. However, it disproportionately favors terms that are easy to place in sentences, can overrepresent frequent terms, and narrows the range of sentence structures and contexts. A high score under this design would therefore show performance on targeted prompts, not general terminology consistency across the domain. Use it for term-level investigation rather than as the primary benchmark.
Of these three designs, the stratified 100-sentence set is the winner. It provides the strongest compromise between breadth, controlled coverage, and interpretable evidence. Before evaluation, pre-register the termbase and the sampling strata, then compute consistency as a document-level exact-match rate: each eligible sentence is judged against the registered term form, and results are also broken out per term. Do not replace this with a token-level average, because weighting repeated or frequent terms by their token count would make common vocabulary disproportionately influential and could conceal term-specific failures.
The build-time check is straightforward: confirm that the set contains 100 sentences, that at least 30 contain domain terms, and that the selected sentences cover the intended frequency bands. The scoring check is equally important: retain the sentence-level match records, inspect every mismatch, and distinguish a wrong term form from a valid contextual variant according to the pre-registered rules. The termbase should be fixed before outputs are scored, with definitions checked against established references such as Wikipedia’s glossary of artificial intelligence, the Luca Malattia terminology and cost guide, and the Scribbr glossary. This process makes the comparison reproducible and keeps the design decision auditable.
Costs and numbers that matter
Token economics make terminology errors more expensive than a simple quality score suggests. Luca Malattia’s Guide to Artificial Intelligence Terminology and Costs, dated March 16, 2026, notes that output tokens can cost up to 5x the cost of input tokens because producing text or code requires active computation. Inconsistent terminology can therefore increase both paid output and the work required to correct the result: a system may consume a draft generation, then consume additional output tokens when editors replace incorrect names with the approved forms. Track generated output tokens and post-generation correction output separately; otherwise, the apparent cost of a terminology failure remains hidden.
The arithmetic behind the acceptance threshold should be explicit, and the denominator must be stated. In the specified 100-sentence, stratified test set, suppose the 30 term-containing sentences average two term instances each. That gives 30 × 2 = 60 scored term instances, not merely 30 scored sentences. With an exact-match check against the pre-registered termbase, 5% of 60 is 3, so the system may have no more than 3 mismatching term instances — this is the instance-level rate used in the worked example below. Recalculate the denominator after evaluation: if those sentences contain 72 instances rather than 60, the allowance changes because the threshold applies to the actual number of term instances. If a team instead applies the threshold at the sentence level, the denominator is the 100 sentences and that choice must be pre-registered; the two rates should not be mixed in one acceptance decision.
Use a simple audit sheet with one row per sentence and one pass-or-fail entry for each expected term instance. A sentence with one incorrect term should not disappear from the count because another sentence contains several correct terms. Summing the failed instances and dividing by all expected instances reveals whether the system actually cleared the threshold. A convenient summary percentage based only on sentences with any terminology error would answer a different question and could conceal failures concentrated in technically dense passages.
The cost of a false positive—accepting a system that falls below 95%—is downstream correction effort. Reviewers must locate every incorrect occurrence, determine the approved replacement, and regenerate or edit the affected output. Because output tokens can be the expensive side of an AI workflow, even a small error count can create a correction loop with visible token and labor costs. The cost of a false negative is different: rejecting a compliant system can trigger unnecessary review or regeneration. A practical rule is to retain the failed-instance log, estimate the correction scope, and compare it with the cost of rerunning the system before making the final acceptance decision.
Term references should help resolve ambiguous labels, not create a competing standard. The Wikipedia glossary of artificial intelligence, the Scribbr glossary of AI terms, and the Malattia terminology and cost guide provide external checks on definitions and cost structures. The operational standard remains the pre-registered termbase: identify the expected term, compare the output exactly, count every mismatch, and report the denominator alongside the percentage. If the audit does not show both 60 expected term instances and no more than 3 mismatches in the stated example, a reported 95% consistency rate should not be accepted.
What the evidence does NOT establish
None of the available sources sources provides a validated 95% threshold for terminology consistency in AI outputs. The threshold should therefore be treated as a method choice rather than an empirically derived cutoff that guarantees acceptable quality. Before using it as a release criterion, teams should record why the chosen level matters for their use case and what consequence follows when the observed score falls below it. A threshold without a documented decision rule is only a reporting convention.
The available references also do not specify how to handle inflected forms or synonyms. The Wikipedia glossary of artificial intelligence, Luca Malattia’s terminology and cost guide, and the Scribbr glossary can help identify terms and establish their meanings, but they do not resolve whether “models,” “model,” or “modeling” should all count as uses of the same concept. The governing termbase must state these rules explicitly: list the preferred form, identify accepted inflections where applicable, and mark each synonym as either interchangeable, prohibited, or requiring a separate count. Ambiguity at this stage can otherwise appear as either an inflated consistency score or an unexplained failure.
There is likewise no variance estimate in the sources for consistency metrics on a 100-sentence test set. Users should calculate uncertainty locally rather than presenting a point estimate as stable. For each relevant term and for the aggregate result, report the number of evaluated sentences, the number meeting the rule, and a confidence interval. A binomial interval is suitable when each sentence is treated as a document-level success or failure; a bootstrap interval is useful when sentences must be resampled within the predefined strata.
The local confidence-interval procedure should be run on several representative test-set compositions, because changing the mix of term-bearing and non-term-bearing sentences can affect apparent precision. Teams should also inspect results by term and by frequency band, checking whether the interval is driven by a small number of common labels or by the less frequent terms. Report failures with their sentence identifiers and term labels so that another reviewer can reproduce the calculation from the registered rules.
Finally, treat the 95% requirement as provisional until the local evidence supports it. Compare observed results with the threshold, examine the confidence interval, and review the substantive errors behind any failed sentences. If uncertainty, missing vocabulary rules, or sparse observations remain unresolved, the score should trigger manual review rather than an automatic pass. This keeps the cutoff operationally useful without implying that the available sources establishes it as a validated standard.
100-sentence test set
This section provides a concrete worked example: a 100-sentence technical-domain test set containing 60 term instances, with three mismatches. Build the set before evaluating the system, and stratify it by term frequency so that sentences with no domain terms, sentences containing one term, and sentences containing multiple terms are represented deliberately. Include at least 30 sentences containing one or more domain terms.
Before scoring, pre-register the approved term forms in a termbase. For every sentence, compare the system’s output with the registered form using exact-match rules, including term boundaries, capitalization, and approved variants. Consistency must be computed as a document-level exact-match rate against that pre-registered termbase, not as a token-level average across all tokens in the corpus. This keeps frequent general words from diluting the result and prevents a high overall token count from disguising terminology errors.
| Checkpoint | Exact matches | Mismatches | Consistency | Decision |
|---|---|---|---|---|
| 1 | 57 | 3 | 57/60 = 95.0% | Meets the 95% threshold |
| 2 | 56 | 4 | 56/60 = 93.3% | Below threshold; reject |
Checkpoint 1 is the passing case: 57 exact matches out of 60 term instances produce 95.0% consistency, so the system meets the threshold. Checkpoint 2 is the rejection case: with four mismatches, 56 of 60 instances match, yielding 93.3%. Because that result falls below 95%, the system should be rejected rather than described as nearly compliant.
The termbase should be informed by named references, not improvised after reviewing system output. Wikipedia’s Glossary of Artificial Intelligence provides definitions of AI terms and concepts, while Scribbr’s Glossary of AI Terms offers additional terminology and acronym definitions. Luca Malattia’s Guide to Artificial Intelligence Terminology and Costs can help identify terminology associated with cost structures, such as input and output tokens. These sources support term selection and interpretation; the pre-registered termbase remains the scoring authority.
Finally, preserve the audit trail. Record the sentence count, the number of term-bearing sentences, total term instances, every exact match, every mismatch, and the resulting calculation. A threshold claim is only defensible when another reviewer can reproduce the count from the same 100-sentence set and termbase without relying on a token-weighted estimate or post hoc substitutions.
Decision rules for terminology consistency
Apply the 95% threshold only after scoring the completed 100-sentence evaluation set, using the instance-level exact-match rule defined above: each expected term instance is scored as a match or mismatch against the pre-registered termbase, and the overall rate is matching instances divided by all expected instances (60 in the worked example). Do not substitute a token-level average, because that can make frequent terms dominate the score and conceal errors in less frequent terminology.
The acceptance rule is conjunctive: accept the output for the domain if overall consistency is at least 95% and every evaluated term has a per-term exact-match consistency rate of at least 90%. The overall rate is the percentage of the 100 sentence-level results that are exact matches. For each term, the per-term rate is the percentage of its applicable exact-match checks that pass. Report the numerator and denominator for every term so the result is auditable and so a term with too few observations is not mistaken for a reliable score.
If overall consistency reaches at least 95% but any term falls below 90% per-term consistency, reject the output. Determine whether the underlying termbase needs revision or whether the system requires additional training. Do not accept the output on the strength of the aggregate score alone: a high overall rate can coexist with inconsistent treatment of a particular domain term, including a less frequent term that matters in the target domain.
If overall consistency is below 95%, reject the output regardless of per-term performance. This rule applies even if most individual terms score at 90% or better, because the document-level requirement has not been met. When a term has no evaluable observations in the 100-sentence set, treat the missing per-term result as an unresolved evaluation gap rather than as a 100% score; correct the test-set or termbase coverage before reconsidering acceptance.
Record the final decision as one of three outcomes: accept when both thresholds are met; reject and revise the termbase or retrain when the overall threshold is met but a term-level threshold is missed; or reject when the overall threshold is missed. Make the decision from the pre-registered scores without changing denominators, excluding failed sentences, or averaging percentages after the fact. Keep the term-level results alongside the overall rate so reviewers can see exactly which condition determined the decision.
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | Pre-register the domain termbase and the required exact-match consistency threshold before testing any AI system. | Pre-registration prevents the acceptance criterion from being changed after results are known. |
| 2 | Build a 100-sentence test set stratified by term frequency, including at least 30 sentences containing domain terms. | The stratified set tests terminology across both common and less frequent usage rather than relying on a narrow sample. |
| 3 | Submit the complete 100-sentence test set to the AI system without revealing the expected termbase forms. | Hidden evaluation preserves a valid check of terminology consistency. |
| 4 | Compare every domain-term instance with the pre-registered termbase and record exact matches at the document level. | Exact matching verifies whether the system uses the approved terms, regardless of grammatical variation or context. |
| 5 | Calculate the document-level exact-match rate by dividing matching term instances by all evaluated term instances, then compare the result with the 95% threshold. | A document-level rate provides one consistent decision measure instead of averaging separate token-level scores. |
| 6 | Accept the AI output for the domain only if the exact-match rate meets the 95% threshold; otherwise, reject it and correct the system before retesting. | Enforcing the same rule for every domain prevents sub-threshold terminology from entering downstream content. |
Frequently Asked Questions
How should the evaluation set be organized before testing?
Build it from 100 sentences and stratify it by term frequency rather than using convenient examples.
What minimum level of domain-term coverage does the test set need?
At least 30 of the 100 sentences must contain one or more domain terms.
How often must each evaluated term appear across the test set?
Every term being evaluated must occur at least three times across the set so each term has enough observations for a meaningful per-term consistency calculation.
When must the terminology rules and termbase be finalized?
The evaluation set must be built before running the system, and the terminology rules must be registered in advance.
What qualifies as a terminology match?
An output occurrence matches only when it exactly matches the preferred form in the pre-registered termbase.
Should consistency be calculated per token or across the whole document?
Consistency should be measured at the document level with exact matching against the pre-registered termbase, not as a token-level average.
Quick answers
| What size should the terminology test set be? | Use a 100-sentence test set. |
| How should the 100-sentence test set be structured? | Build 100 sentences and stratify them by term frequency. |
| What minimum number of sentences must contain domain terms? | At least 30 of the 100 sentences must contain domain terms. |
| What criterion must outputs meet under the 95% Terminology Threshold? | At least 95% must meet the pre-registered terminology criterion. |
| How should terminology consistency be measured? | Measure consistency at the document level with exact matching against a pre-registered termbase, not a token-level average. |
Also worth reading: How AI Translation Models Handle Ancient Hebrew Praise Terminology A Technical Analysis of Religious Text Translation Accuracy: How AI Translation Models Handle · Understanding Mexican Scallop Terminology AI-Powered Translation Guide for Seafood Industry Terms: Understanding Mexican Scallop Terminology AI-Powered · The Impact of Migration Terminology on AI Translation Accuracy Emigrate vs Immigrate in 7 Languages: Impact of Migration Terminology on