Practices for Automated Benchmark Evaluations of Language Models

Record US-21 · NIST AI 800-2 ipd; DOI 10.6028/NIST.AI.800-2.ipd

Issued by: Center for AI Standards and Innovation (CAISI), National Institute of Standards and Technology (NIST)
Jurisdiction: United States
Level · region: National · North America
Published / adopted: 2026-01
Language: English
Official identifier: NIST AI 800-2 ipd; DOI 10.6028/NIST.AI.800-2.ipd

Summary

Draft voluntary best practices for automated benchmark evaluations of language models and AI agents, covering how to define objectives and choose benchmarks, how to implement and run evaluations (including debugging cheating and evaluation awareness), and how to analyse and report results with sound statistics and qualified claims.

Safety topics

evaluation & testingsafety institutegeneral-purpose / frontier AI

Status notes

Initial public draft dated January 2026; NIST invited public comments until 31 March 2026. NIST's guidelines page, updated 30 September 2026, still lists only the initial public draft, now under the Center for Advancing Innovation and Standards for Super Intelligence (CAISSI), the name NIST uses for CAISI after EO 14434 (US-26). The draft answers the AI Action Plan's call (US-16) for CAISI to publish guidelines and resources for federal agencies' evaluations of AI systems, and refers to NIST AI 100-1 (US-02) and NIST AI 800-1 (US-07) for organisational evaluation practices.

Sources

Open the official source https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf

Secondary source: https://www.nist.gov/caissi/guidelines

Local copy

Archived. Last attempt 2026-10-17 22:12:14 UTC

Official source (third-party copy, not published) archive/US-21/2026-10-11/official.pdf

Retrieved 2026-10-11 14:04:01 UTC

SHA-256 5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7

Secondary source (third-party copy, not published) archive/US-21/2026-10-11/secondary.html

Retrieved 2026-10-11 14:04:01 UTC

SHA-256 e7e2e49a29159080c3dfaac77fe2eafe9d54b206c1134d104e3ab020bd3072b4

Verification

The instrument's own official text, or an official register, was stored and confirms the title, date and status. checked 2026-10-18 · official-url-fetched

Recheck of 17 Oct 2026: re-read the nvlpubs.nist.gov PDF and the NIST guidelines page stored on 11 Oct 2026 (the archiver found both unchanged on 17 Oct 2026). The PDF confirms the title, NIST AI 800-2 ipd, the DOI, January 2026, CAISI authorship and draft status; the guidelines page (updated 30 Sep 2026) still lists only the initial public draft and gives the 31 Mar 2026 comment deadline. The earlier notes' description of the draft as CAISI's 'main published guideline after the 2025 reorganisation' was replaced with what the page shows.

Second check: Independent check by Claude Code (claude-sonnet-5-5), 2026-10-18: 8 field(s) supported.

History

2026-10-18notes: Initial Public Draft released January 2026; public comment closed 31 Mar 2026. N → Initial public draft dated January 2026; NIST invited public comments until 31 M — rewritten: the guidelines page of 30 Sep 2026 still lists only the draft, under the renamed CAISSI; the unsupported 'main published guideline' remark removed2026-10-17-recheck-north-america-1 · Claude Code (claude-opus-5-5)
2026-10-18related: [{"slug": "us-caisi", "id": "US-15"}, {"slug": "us-nist-ai-800-1", "id": "US-07" → [{"id": "US-15", "slug": "us-caisi"}, {"id": "US-07", "slug": "us-nist-ai-800-1" — US-16 added (the draft answers the Action Plan's call for evaluation guidelines) and US-02 added (the draft refers to NIST AI 100-1 for organisational practices)2026-10-17-recheck-north-america-1 · Claude Code (claude-opus-5-5)

Related records

Center for AI Standards and Innovation (CAISI), now CAISSI United States · Published (non-binding, current)

NIST AI 800-1 United States · Draft / proposed

America's AI Action Plan United States · Published (non-binding, current)

NIST AI RMF United States · Published (non-binding, current)

Citation

Center for AI Standards and Innovation (CAISI), National Institute of Standards and Technology (NIST). Practices for Automated Benchmark Evaluations of Language Models [NIST AI 800-2 ipd; DOI 10.6028/NIST.AI.800-2.ipd]. 2026-01. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf. Status: Draft / proposed; verified 2026-10-18 (level A). Local copy archive/US-21/2026-10-11/official.pdf retrieved 2026-10-11 14:04:01 UTC, SHA-256 5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7. Every AI Policy Ever, version 1.18 (snapshot 2026-10-18), record US-21.

Improve this record

Know of an amendment, a status change or an error? Report news or an error on GitHub; the form opens with this record filled in.