Context counts: Measuring how AI reflects local realities in education

If an AI tool in rural Tanzania generates a lesson plan with pizza instead of chapati, it's already failed. Context counts!

And evidence suggests less than 0.2% of the data that AI models are trained on comes from Africa and South America.

Without context, AI models fail to produce good examples for low- and middle-income countries or link to the national curricula - something we hear frequently from our community.

This challenge is referred to as “contextualisation”, which we define as the extent to which an output is adapted to make it relevant to a specific audience.

As teachers and children are increasingly using AI generated materials there is a need to have more scalable judgements of content.

At Fab AI, we are working on a range of AI benchmarks (such as for pedagogy, and visual reasoning) and evaluations to help product developers and consumers choose the best models and to highlight to big tech where they can improve.

In this report we codify what we mean by contextualisation and review the current landscape to identify what existing benchmarks are available to test if AI generated content is suitable for users' educational contexts.

See our benchmarks leaderboard for up to date model performance against our AI for education benchmarks for LMICs and download the paper below to find out more about contextualisation.