Researchers from the University of Colorado (Anirudh Maiya, Razan Alghamdi, Maria Leonor Pacheco, Ashutosh Trivedi, Fabio Somenzi) tested LLMs on 6×6 Sudoku. Recent studies show that modern language models can solve logical problems but are still unable to clearly explain their reasoning:
- The study used 2,293 unique puzzles.
- Five models were tested: four open-source (Gemma, Mistral, and two versions of Llama) and one closed model from OpenAI (o1-preview).
- The tasks ranged in difficulty from easy to “diabolical.”
Results:
- Open-source models solved less than 1% of the puzzles, while OpenAI performed much better – with 65% correct solutions.
- For simple puzzles (Easy and Medium categories) OpenAI showed 100% accuracy. For the most difficult Sudoku (“Diabolical”), accuracy dropped to 40%.
Limits of AI capabilities
The authors of the study examined how AI explains its steps. They selected 20 puzzles and asked experts to evaluate the answers according to three criteria:
- Justification. Only in 5% of cases was the model able to explain why it chose a particular number. In the rest, it merely listed Sudoku rules unrelated to the specific puzzle.
- Clarity. Only 7.5% of explanations were clear and coherent; the others relied on vague statements, contradictory terminology, or skipped steps.
- Practical value. Only 2.5% of explanations helped understand the actual solving strategy.
Why answers alone are not enough
AI is increasingly used in medicine, business, and law. But if a model cannot transparently explain its reasoning, its use in critical areas becomes risky. In such cases as diagnosis or legal decisions, explanations may be more important than the answer itself. The researchers note that for models to become truly useful partners, they must be trained not only to find correct answers but also to translate complex reasoning into language understandable to humans.
