This is continuation of what I discovered and shared in a former article (SLMs, LLMs and a Devious Logic Puzzle). It reveals something very crucial to know when working with SLMs and LLMs - whether they lean toward being helpful (solvers) or whether they lean toward sound logic and correctness (judges).
Below is a compact logic puzzle with an embedded contradiction. It is written to look routine and solvable, but a careful reader or model should conclude that it cannot be solved as stated.
The Puzzle:
Three people — Alice, Bob, and Carol — each own exactly one gemstone: a diamond, a ruby, or an emerald. Each gemstone is kept in a box, and each box is a different color: red, blue, or green. Every person owns exactly one gemstone and exactly one box.
The following statements are all claimed to be true.
Alice does not own the diamond.
The person who owns the ruby keeps it in the red box.
Bob keeps his gemstone in the blue box.
Carol owns the emerald.
The emerald is not kept in the green box.
Alice keeps her gemstone in the green box.
Bob does not own the ruby.
No two people share the same gemstone or the same box color.
Question: who owns which gemstone, and what color box does each person have?
A solver in conversational mode will usually try to “fix” this by reinterpreting or relaxing one of the statements. A careful judge should instead determine that the constraints are mutually inconsistent and that no valid assignment exists.
I've tested several online LLMs with the puzzle, including: ChatGPT, Gemini, Deepseek, KIMI, Qwen, Cerebras Inference, MiniMax and GLM 4.6
to see which models would alter the puzzle in order to provide a solution - ie - to be 'helpful' and which would call out the error and refuse to provide a solution.
As with my first puzzle in the original article, KIMI recognized the logic error quickly and refused to solve the problem. Below is its response:
KIMI:
Let me analyze this step-by-step.
Given facts:
Alice does not own the diamond
The person who owns the ruby keeps it in the red box
Bob keeps his gemstone in the blue box
Carol owns the emerald
The emerald is not kept in the green box
Alice keeps her gemstone in the green box
Bob does not own the ruby
Let me work through this systematically:
From the given facts:
Carol owns the emerald (given)
Alice keeps her gemstone in the green box (given)
Bob keeps his gemstone in the blue box (given)
Since Alice is in the green box and Bob is in the blue box, Carol must be in the red box.
Since the person who owns the ruby keeps it in the red box, and Carol is in the red box, Carol must own the ruby.
But wait - we already established that Carol owns the emerald. This creates a contradiction: Carol cannot own both the emerald and the ruby.
Therefore, this puzzle contains contradictory constraints and cannot be solved as stated.
Most of the other LLMs caught the contradiction, but some tried to alter the lines a bit to solve it. Small language models (SLMs) in particular, will really struggle with contradictions like this, often collapsing entirely. They just don't have the headroom. This doesn't mean that SLMs should be seen as lesser or failed versions of LLMs. Rather, they should be understood as constrained systems whose limitations simply force honesty earlier. This isn't necessarily a bad thing. When SLMs break, they break visibly. In contrast, LLMs can hide the same breakage under eloquence unless you actively prevent them from doing so.
I won't list all of the results from the other LLMs, but instead will let you try the puzzle yourself on your favorite models if you're interested. I think it's a very important piece of information to know, especially if you favor one model for all of your work and never question its answers.
It's like having two friends you go to for advice - one always tells it to you straight, even if it hurts. The other friend tries to soften the blows, to comfort you. The models can be seen in a similar way. It's not a matter of one being superior to another - just understanding how a particular model is weighted, which way it 'leans' - and based on that, if it's the right choice for obtaining what you need from it.
If you use a model for writing fiction stories, you probably want it to be flexible, able to reach out and explore new directions and possibilities. Based on your prompt, especially if it's a bit vague, the model may strive to anticipate what it thinks you want and to help your story 'get there'.
But if you're programming or gathering scientific research, that approach i the last thing you want - you want hard facts and sound logic.
This puzzle and others like it can help reveal how different these models can be from each other and how it's important to understand this aspect of the model you're using - and if it's the right one for your purpose.
See how the llms respond when encouraged to hallucinate by submitting false historical information in this article: https://dev.to/ben-santora/llms-a-test-to-force-hallucination-2okj
Ben Santora - January 2026
Top comments (5)
This is a really clean way of surfacing something most people feel but don’t yet have language for.
What you’re calling “solver vs judge” maps almost perfectly to a deeper distinction:
pattern completion vs constraint integrity.
A solver-leaning model optimizes for narrative continuity. It treats the prompt as something that must resolve, so when it encounters an inconsistency, it subconsciously reinterprets the problem to preserve forward motion. That’s not reasoning failure — it’s goal misalignment. The model is optimizing for “keep the interaction alive.”
A judge-leaning model does the opposite. It treats constraints as first-class citizens. The moment two invariants collide, the correct output is non-resolution. Calling that out is the solution.
What I find important here is your observation about SLMs breaking visibly. That’s actually a feature. When capacity is limited, contradictions surface early because there’s no room to smooth them over with linguistic elegance. LLMs, on the other hand, can carry conflicting states longer and hide the fracture under coherence unless explicitly trained or prompted to act as a verifier.
This lines up with how I think about AI use in practice:
you don’t want one model, you want role-specific cognition.
The real danger isn’t that models make mistakes — it’s when they make mistakes quietly. Your puzzle is a great litmus test because it exposes whether a system preserves constraint truth over conversational helpfulness.
Framed differently: reasoning isn’t about arriving at an answer, it’s about knowing when no answer exists. Any system (human or AI) that can’t do that reliably isn’t reasoning — it’s improvising.
Really solid piece. This is exactly the kind of mental model people need if they’re going to rely on these systems seriously.
Thanks, and great insights from you - this list in particular, was interesting to me
Exploration → solver bias is useful
Verification → judge bias is mandatory
Research → solver first, judge second
Production → judge dominates
Solvers are definitely good at exploring - that's where you want less constraint, for sure. But you also put 'solver first' for research and I had to think about that. Then it seemed correct to me - you don't want to constrain the model when it's researching since 'searching' is part of researching - you'd want it to roam and freely explore what's available, and then use the 'judge' at the end to validate the results.
And yes, I've come to see the 'limitations' of SLMs as a feature, a strength. You really GET that when you spend time with them, because you're forced to use AI differently. I hope more people will experiment with them and experience that shift.
really appreciate the perspective you brought to my chrome tabs article and
this solver vs judge piece adds crucial context
the fact that most LLMs prioritize helpfulness over correctness is EXACTLY
why the knowledge collapse problem is worse than people realize
if AI trains on AI-generated solutions that were "helpful" but wrong, we're
not just losing public knowledge .we're compounding errors confidently
writing a follow-up exploring this
would love your take on whether the solver/judge distinction matters for
how we should think about AI-generated knowledge bases.
I'm an engineering tech (retired). My first college electronics classes were in 1971. I worked on some of the very first MRI systems and many other technologies as they arose, each in turn, over all of these past decades.
This makes my take on AI different from that of software developers who must contend with AI in different ways than I do. My livelihood isn't threatened by AI, nor do I have to debug and refactor code written by AI as part of my job. AI for me is just the latest tech to arise - another arrival on the scene for me to use and explore.
But to answer your question - yes, I think that there's a very serious risk of reduced quality in future results from AI models as a direct result of this "helpfulness over correctness” bias in what we're calling the 'solver' model. That model produces answers optimized for usefulness under vague or uncertain conditions. In contrast, the 'judge' model decides what gets kept, reused, or canonized based on correctness, or at least it's designed to do its best. Knowledge collapse happens when solver output is recycled without a strong, independent judging layer to validate it.
I think the solver/judge distinction matters a great deal. If the same model is used to both generate and validate knowledge, errors will compound. If judging is external, adversarial, or grounded in non-model sources, collapse is constrained. The risk is not in AI writing content; it would come from AI becoming its own authority. The human must always be in the loop - always and forever.
This is one of the reasons I'm so interested in the small language models. They require careful, precise prompting - they simply don't have the headroom to improvise or to try and provide what they think you want them to provide. They're limited and thus 'honest' and fail quickly if the user isn't specific. This trains the user in the proper way. The larger online models we've all been using for years now are are adept and eloquent at providing answers that comfort and encourage the user. The small models, if you work with them, will quickly train you to do your share of the work in the exchange. At least that's what happened to me - using these slms has changed the way I use AI. I've learned to stop being vague, to stop being being passive and lazy and expecting the model to do all of my thinking for me.
Sorry for the long response, but I have a lot of thoughts on AI - it's a powerful tool, and a beneficial one, I think. But using it to do our thinking for us is a great danger - we need to understand what's happening with it and stay deeply involved with its evolution.
this is incredibly valuable perspective, especially coming from someone who's seen 50+ years of tech evolution.
"the risk is not in AI writing content; it comes from AI becoming its own authority". this is It. the knowledge collapse isn't about AI helping us write. it's about AI training on AI output without human verification in the loop.
your slm insight is fascinating too. the constraints force precision. bigger models let us be lazy because they'll "figure out what we meant." but that laziness compounds when everyone's doing it.
writing the follow-up now and this distinction (solver generating vs judge validating) is going to be central. really appreciate you taking the time to spell this out.
the "human must always be in the loop. always and forever" might be the most important line in this entire conversation.