The new NIST program aims to provide more objective AI model testing by evaluating systems on blind datasets using common metrics.
The National Institute of Standards and Technology (NIST) has launched a new Artificial Intelligence Technology Evaluation (AITE) program that will provide researchers and developers with a sequestered testing environment to evaluate artificial intelligence (AI) models on blind datasets across multiple domains.
According to a July 27 NIST bulletin, the program is designed to provide more rigorous and objective assessments of AI model performance by preventing test data from being incorporated into model training.
The initiative is being led by NIST’s Technology Test and Evaluation Division and will initially evaluate large vision language models (VLMs) on tasks spanning quantum science, genomics, and public safety. Additional evaluation tasks are planned over time.
“The infrastructure provided by NIST will provide common data, metrics, and scoring to help developers understand the performance of their models,” NIST said.
The program includes two participation tracks.
Under one track, data providers can submit original datasets and associated evaluation tasks in their areas of expertise. NIST said those participants will receive performance measurements of leading AI models evaluated against their datasets.
Under the other track, model providers can submit AI models for testing against the growing collection of datasets and tasks. NIST said participants will receive information on how their models perform across multiple domains and how those results compare with other models using the same metrics “while ensuring the evaluation data is not used for training any model.”
Participation is open to organizations and individuals that agree to the program’s participation agreement and rules. NIST said interested participants can request participation or ask questions by contacting the agency at aite-poc@list.nist.gov.
For more information, prospective participants can check out the agency’s overview of the AITE program.
