Visual Reasoning Benchmark

We took our work on visual maths further, using questions from Zambia and India to test AI models on non-verbal reasoning tasks. We made The Visual Reasoning Benchmark to test if AI models can answer genuine visual questions faced by end-of-primary students in LMICs.

Our Visual Reasoning Benchmark incorporates multiple choice questions from end of primary non-verbal, cognitive aptitude assessments used in Zambia and India. These questions differ from those in our Visual Maths Benchmark which consider maths concepts represented with images, as they don't rely on language or numerals. Instead, they focus on visual tasks such as pattern recognition, matching and spatial reasoning, like the well-known Raven's Progressive Matrices.

We're using this benchmark to better understand how well AI models can tackle non-verbal visual tasks. This helps pinpoint where they can be relied on for educational support, where they're likely to fail and what still needs work.