New randomized, controlled trial by the World Bank of students using GPT-4 as a tutor in Nigeria. Six weeks of after-school AI tutoring = 2 years of typical learning gains, outperforming 80% of other educational interventions. And it helped all students, especially girls who were initially behind. - ThreadSky

emollick.bsky.social • 42 days ago

New randomized, controlled trial by the World Bank of students using GPT-4 as a tutor in Nigeria. Six weeks of after-school AI tutoring = 2 years of typical learning gains, outperforming 80% of other educational interventions.

And it helped all students, especially girls who were initially behind.

1 / 2

Comments

nathanfavero.com•42 days ago

Sounds promising, but we can agree the estimated effect size is totally implausible, right?

emollick.bsky.social•42 days ago

.3 SD?

nathanfavero.com•42 days ago

Which is supposed to be 2 years of typical learning gains. From a 6 week program. Isn’t that a bit implausible?

mjfgates.bsky.social•42 days ago

So this is the second best, out of six alternatives? What was the actual best one?

thedevlinb.bsky.social•42 days ago

To be clear, the program had a teacher leading the class. Comparison is to students with no extra tutoring. With extra hours of practice there is no surprise that the students' English was better. A comparison should be done to similar curriculum that is not AI assisted but still using edtech.

robb.doering.ai•42 days ago

Hmm I'm not sure I understand the critique -- what "EdTech" are you saying is missing? Like, giving the students paid access to some platform? Or are you just saying that the study should've controlled for total hours spent overall per student, and it didn't?

robb.doering.ai•42 days ago

"Each additional day of attendance resulted in significant improvements in learning outcomes."

Yeahhh seems like a valid critique. The response, ofc, would probably be that without AI tutors, the classes would be too expensive to run and wouldn't happen in the first place

kfet.dev•42 days ago

Nope, that's inaccurate, they actually compare results with a broad study of different educational interventions, including teacher professional development.

thedevlinb.bsky.social•42 days ago

They compared their results to other studies, informative, but not the same as a like to like from the same population. I love what AI can do for learning (I use it all the time to help me read journal articles!), but I also love mathematical rigor!

kfet.dev•42 days ago

There is nothing unsound in comparing the effects to other studies, that's the whole point of measuring those effects.

It's a valid result in the most strict mathematical rigour. As far as we can tell from the blog post, that is.

kfet.dev•42 days ago

Let's keep in mind that this is not a formally designed experiment in a lab, instead this is a program designed to help actual students out there get better educated.

And it did. By a lot!

thedevlinb.bsky.social•42 days ago

Also this is a highly controlled ed-tech setup. Ed-tech interventions often have great results in initial pilot studies because the instructors are well trained. (As they were here). Going from a test class to mass deployment in the real world is *hard*. Lots of post-mortems on this topic!

thedevlinb.bsky.social•42 days ago

The improvements are impressive, but they cannot be 1:1 compared to other methodologies, especially given the insane diversity of schooling and even living conditions in the countries being tested.

kfet.dev•42 days ago

One interesting conjecture (not studied) in the article:

"This suggests that students who learned to engage effectively with AI may have leveraged these skills to explore and master other topics independently."

If true this is really powerful, learning on your own is the single best way to retain.

buildit.bsky.social•42 days ago

So much of the AI discourse here has been focused on AI in (American) university classrooms, which makes sense. This is where many ppl developing these tools work! But the real exciting (short/medium term) opportunity IMO is in expanding access as in this study

outeast.bsky.social•41 days ago

📌

siriusli.bsky.social•41 days ago

I would love to see a repeat of the Hole in the Wall project but with an LLM.

https://www.ted.com/talks/sugata_mitra_the_child_driven_education

mustafasuleymanai.bsky.social•42 days ago

So powerful. This is exactly why I'm so passionate about AI—opening up access and opportunity for people. Even more astounding when you take into account that many of these students had never even used a computer before this pilot. Gets me so excited for what's possible, for everyone.

boundaryspanner.bsky.social•42 days ago

It will be interesting to see longer-term effects, especially in general cognitive abilities. Fen-phen is a really effective weight loss drug in the short term.

Other studies are starting to indicate negative effects students of using AI on critical thinking and other brain development markers.

emollick.bsky.social•42 days ago

They used Microsoft Copilot and teachers provided guidance and initial prompts. No working paper yet, but the results and experiment are written up here: https://blogs.worldbank.org/en/education/From-chalkboards-to-chatbots-Transforming-learning-in-Nigeria

eduglobe.bsky.social•42 days ago

This is quite intriguing - I would love to see their prompts to understand better what they did. Unfortunately, I could not find these prompts. Do you have more information about them?

esgarchitect.bsky.social•42 days ago

Scaling good teachers. They deserve a cut of the chat bot fee.

emollick.bsky.social•42 days ago

And since it isn't clear to everyone who doesn't read the blog post - the fact that this is teacher-led is likely pretty important. We know that independent student use of AI as a tutor can harm learning in some circumstances, because it gives the illusion of learning.

emollick.bsky.social•42 days ago

I wrote more about when AI helps and hurts education in this earlier post. https://www.oneusefulthing.org/p/post-apocalyptic-education

kfet.dev•42 days ago

They link to an earlier post from back in September, with seven key learnings from the program. That's before they had these amazing results to report on, but it's still very informative:

https://blogs.worldbank.org/en/education/From-chalkboards-to-chatbots-in-Nigeria

kfet.dev•42 days ago

1. Prioritizing students: high engagement and enthusiasm from students
2. Inspiring teachers: teachers overcame initial AI concerns
3. Optimizing immersion: longer program duration needed
4. Nurturing infrastructure: reliable power/internet crucial

kfet.dev•42 days ago

5. Empowering with materials: toolkits helped guide AI interactions
6. Enhancing implementation: close monitoring ensured success
7. Reducing AI risks: addressed concerns like over-reliance and misinformation

kfet.dev•42 days ago

Tangential: yes I used an AI to summarize those. I did read the whole article and reviewed it all, and still found it a useful tool.

pmwwp.bsky.social•42 days ago

Well damn.

catblanketflower.yuwakisa.com•42 days ago

AI tutoring is extremely gentle

sinez15.bsky.social•41 days ago

Thanks for sharing!

unremarkable.ai•42 days ago

Totally expected. Also, why teach when all knowledge can be on-demand?

I think the "When the Bough Breaks" TNG episode was the first time I saw Sci-fi think about that. The AI summary of it in this context from o1:

unremarkable.ai•42 days ago

"Blindly relying on advanced technology stifles genuine learning, reminding us of the critical need for foundational knowledge and continuous inquiry."

bramzijlstra.com•42 days ago

That’s awesome. LLMs have been very useful for interview prepping for me last year, I can see it being helpful for school work as well

georgethebaker.bsky.social•42 days ago

As someone in an industry that requires technical tacit skills this concerns me deeply.

I see less and less basic nervous
Mechanical and physical dexterity yearly with interns and work studies.

That lack of repetitive motion to memory capture is growing and problematic when things physical break

rushhour.bsky.social•42 days ago

- massive handwriting over years Sint done anymore, Pupille loosing eye-Brain-Hand coordination

- school concentrates on intellectuall learning, while sports, Music and Arts (with more physical Elements: catch a ball, paint, carve, play Instrument etc) are downsized more and more

markwarschauer.com•42 days ago

Looks promising but I’m surprised that the World Bank would release important findings via a blog post without even a preprint or research brief

sorrykb.bsky.social•32 days ago

Almost like the findings are ridiculous and unprovable

Comments

Posting Rules

Reply