Is an AI “smart” if it meets certain benchmarks? How much does it “know”? Can it really “think”? Answering these questions has led to a proliferation of benchmarks measuring different capacities. The sheer quantity poses challenges. Moreover, benchmarks often point in different directions, making it difficult to compare models.

This talk will walk through distinctions that help us make sense of the expanding and complicated landscape of benchmarks. In particular, we will do a deep dive on the distinction between knowledge and intelligence, using the Abstraction and Reasoning Corpus (ARC) as an example, as well as a home-grown version that introduces metaphors (MARC).