AI Agent Assessment: From 'College Entrance Examination' to 'Performance Evaluation', A New Paradigm for AI Assessment

Employee evaluations typically encompass three main dimensions: "performance", "behavior", and "professional ethics". AI agent assessment can also be divided into result assessment, process assessment ...
TMCnet

MITRE and FAA Introduce Novel Aerospace Large Language Model Evaluation Benchmark

The Federal Aviation Administration (FAA) and MITRE are introducing a new benchmark to enable the evaluation and assessment of large language models (LLMs) for aerospace tasks. Given the ...
TechCrunch

OpenAI’s o3 AI model scores lower on a benchmark than the company initially implied

A discrepancy between first- and third-party benchmark results for OpenAI’s o3 AI model is raising questions about the company’s transparency and model testing practices. When OpenAI unveiled o3 in ...
JSTOR Daily

Examining Rater Errors in the Assessment of Written Composition with a Many-Faceted Rasch Model

This study describes several categories of rater errors (rater severity, halo effect, central tendency, and restriction of range). Criteria are presented for evaluating the quality of ratings based on ...