Abstract

Conventional AI benchmarks typically focus on the content of responses, for example checking factual (e.g. MMLU ) or mathematical correctness (e.g. GSM8k). However, for many language model applications, the manner (or “personality”) of a model’s responses also matters to users, for example how friendly or confident responses are. Recent issues with model releases highlight the limited ability of existing evaluation approaches to capture such personality traits: a ChatGPT model version was rolled back over sycophant personality issues, other models’ personalities have been critised to overfit to the Chatbot Arena leaderboard.

In this talk, I will introduce Feedback Forensics: our newly released toolkit to measure AI personality traits. Using our toolkit, I will first share results detecting the personality traits currently encouraged by popular human feedback datasets (incl. Chatbot Arena). Next, I will discuss changes and trends in personality traits exhibited across model families and versions. Finally, I will take a closer look the personality differences between the Chatbot Arena and publicly released version of Llama-4-Maverick.

The talk will feature a live demo of our personality visualisation tool and attendees are invited to follow along via our online platform https://feedbackforensics.com/ (laptops are encouraged).

You can also join us on Zoom