See Selected Projects

Education & skillsBrazil

Give students instant AI feedback on practice essays, and teachers more time to discuss them

In Espírito Santo, Brazil, public school seniors practise for the entrance exam essay on Letrus and get instant AI feedback. In a randomised trial in 178 schools, students given access scored about 17 points higher out of 1,000.

Try it withClaudeChatGPT / OpenAI modelsGeminiGrok

Deployed; tested in a randomised trial in 178 Espírito Santo state schools in 2019 and offered to all senior-year students in the state's schools since 2022, Letrus, with the Espírito Santo State Education Department (Sedu), Brazil

Why it matters

Brazil's national university entrance exam, ENEM, includes an essay, and J-PAL notes that the gap between public and private school students is larger on the essay than on the other sections. Letrus, a Brazilian education technology company, built a platform that marks practice essays against the exam's criteria and returns feedback at once; J-PAL notes that teachers often lack the time to give every student that kind of individual feedback. In 2019 Espírito Santo's State Education Department (Sedu) tested the platform in a randomised trial with researchers Bruno Ferman of the São Paulo School of Economics, Flavio Riva and Lycia Silva e Lima. About 12,000 students in 110 schools got access and 68 schools did not. Students with access scored about 0.09 standard deviations higher on the entrance exam, about 17 points out of 1,000, which closed about nine percent of the public-private gap in essay scores. A version that added comments from human scorers three days later did no better than AI feedback alone, so in 2022 the state chose the AI-only version, which J-PAL says reached similar results at lower cost and gave prompt feedback, and offered it to all senior-year students in its schools. J-PAL reports more than 100,000 seniors enrolled since 2020, including over 28,000 in 2024.

How it works

A Portuguese teacher leads the class through a writing activity (two a month under Sedu's 2022 plan) and students draft their essays from support texts. Students then post their essays on the platform, which marks them against the ENEM criteria and replies at once; in the trial, the AI feedback covered style and syntax. Sedu's 2020 notice says students see feedback on the structure of the text, their performance on each ENEM competency and a score from 0 to 1,000. The teacher reads the platform's feedback, decides whether to amend its comments and gives the student the final feedback, and can follow the progress of each student and each class. In the trial, students with access wrote more practice essays and discussed more of them one to one with their teachers, which J-PAL takes as a sign that the platform saved teachers grading time. When UNESCO named Letrus a winner of its King Hamad Bin Isa Al-Khalifa Prize in May 2020, UN News described software feedback combined with more detailed human corrections; Espírito Santo uses the AI-only version. Sedu's July 2024 release says second- and third-year classes use the tool in lessons and that first-year classes would join from 1 August 2024. J-PAL says Mato Grosso do Sul and Alagoas adopted the platform in some schools in 2021, Paraíba and Pará in 2022, and Mato Grosso and Goiás through direct partnerships in 2024.

Who it's for

State and city education departments, school networks and exam preparation programmes that want students to write more and get feedback on every essay, and researchers planning trials of AI feedback in schools.

The prompt

You are helping me plan a pilot in which students get instant AI feedback on practice essays and a teacher reviews that feedback before the student sees the final version. Do not ask me for any real student's name, essay or marks.

Our school system: [for example, 40 public secondary schools, 6,000 final-year students, one language teacher for every 150 students]
What the essays prepare for: [for example, the essay section of our national university entrance exam, with its published marking criteria]
What we can use: [for example, an AI model under a contract that protects student data, and school laptops for two lessons a week]

1. Turn our marking criteria into a rubric the AI can apply, with a plain description of each level and one example comment per level. Leave placeholders where the official criteria are not public.
2. Write the rules the AI must follow: comment on the student's own text, quote the passage each comment refers to, give one next step per criterion, never rewrite the essay for the student, and send any essay that mentions harm, abuse or distress straight to the teacher without scoring it.
3. Describe the teacher's step: how a teacher reviews, edits or rejects the AI feedback for a class of [number] students in [minutes], and what the teacher sees about each student and the class.
4. Set privacy rules: what student data is stored, where and for how long, who can see it, and whether essays may be used to train any model.
5. Design a randomised evaluation: schools assigned by lottery to AI feedback, AI feedback plus human scorers, or no access; the outcomes to measure (exam essay score, essays written, essays discussed with a teacher); and how to check a sample of AI scores against trained human markers.
6. List the cost per student of each version and the result that would justify offering it to every school.

Leave placeholders rather than invent rules, figures or targets, and mark everything a school leader or education department must sign off before launch.

Open inClaude (opens in a new tab)ChatGPT (opens in a new tab)Gemini (opens in a new tab)Grok (opens in a new tab)

Sources

The three Sedu releases are from the state department that runs the programme, and the July 2024 release was written with information from Letrus's press office. The trial was run by a J-PAL-affiliated researcher in partnership with Letrus and Sedu and funded by Google.org through J-PAL, so J-PAL's two pages are not fully independent of the programme. The working paper is by the trial's university researchers and was not found in a peer-reviewed journal. UN News reports UNESCO's prize. No independent check of the AI's marking accuracy was found.

  1. 1. J-PAL (Abdul Latif Jameel Poverty Action Lab): "Both interventions increased students’ entrance exam scores by about 0.09 standard deviations". Artificial intelligence to strengthen high school students’ writing skills (opens in a new tab)

  2. 2. J-PAL (Abdul Latif Jameel Poverty Action Lab): "adding trained graders to review AI-generated feedback on student essays didn’t improve student outcomes". An AI assistant for teachers (opens in a new tab)

  3. 3. Stanford SCALE AI repository (working paper by Ferman, Lima and Riva): "Both technologies significantly improved students' essay scores in a large college admission exam". Artificial Intelligence, Teacher Tasks and Individualized Pedagogy (opens in a new tab)

  4. 4. Government of Espírito Santo (Sedu): "todos os estudantes da 3ª série do Ensino Médio da Rede Estadual". Estudantes da 3ª série do Ensino Médio passam a contar com Plataforma Letrus (opens in a new tab)

  5. 5. Government of Espírito Santo (Sedu): "devolutiva instantânea sobre a estrutura do texto, com o desempenho em cada competência". Estudantes podem participar do Programa Letrus de Desenvolvimento da Escrita (opens in a new tab)

  6. 6. Government of Espírito Santo (Sedu): "o uso da Inteligência Artificial fomentou o trabalho desenvolvido pelas equipes escolares". Mais da metade dos alunos da Rede Estadual do Espírito Santo tem nota acima de 600 na redação do Enem 2023 (opens in a new tab)

  7. 7. UN News: "usa la inteligencia artificial para proporcionar correcciones automáticas de la escritura de los estudiantes". La UNESCO premia proyectos de Brasil y España de uso de la inteligencia artificial en la educación (opens in a new tab)

Related projects