Using a variety of analogy puzzles, SFI researchers have shown that the reasoning abilities of OpenAI’s GPT-4 model fall short when faced with small changes. Recognizing the limitations of these tools is critical to knowing when and to what extent they can be trusted.

https://santafe.edu/news-center/news/study-large-language-models-still-lack-general-reasoning-skills