Notes on Apple's reasoning illusions arguments

Evaluation, Data Contamination, and Reasoning Models Analysis

• Evals based on final answer accuracy

  • Math and code are fields where LLMs are seen the strongest

• Data contamination - what is it?

  • Affects eval results

• Two things for reasoning models analysis :

  • How complex is problem
  • How logical is the LLM reasoning structure

Scaling, Complexity, and Performance

• Problem exposed - frontier LRMs have a “collapse” in accuracy beyond certain complexity

• There is also a scaling limit - what is scaling specifically here?

• Reasoning effort inversely proportional to problem complexity, but then decline after, no plateau?

  • Giving Up essentially?

• LLMs actually outperform LRMs in low complex problems (overthinking)

• LRMs superior (due to their additional thinking) in medium complexity

• High complex problems both models collapse

Limitations, Reasoning Traces, and Reasoning Behavior

• The limitation of LRMs - dont use directly accessible algorithms and fail

• Reasoning traces - go more deeper into this

• Essentially wanna answer if models like 03, claude 3.7, gemini thinking are really reasoning at all

• Common thinking by LRMs - Chain of thought

  • Self reflection

• When LRMs fail a complex problem - fixate on an early wrong answer (chain of thought continues this, and the rest of the tokens wasted spending time on this)

  • How can LRM’s know when to double check or when to stop?
  • Also, they might know right answer, but for some reason use up tokens to explore incorrect alternatives to be safe - overthinking?
  • Both these issues waste tokens

• Is reasoning actually logical assessment or just different forms of pattern matching?

• Failure analyses as important as providing reasoning for the answer is right

• More algorithmic reasoning needed than just a different form of pattern matching

Intelligence, Data, and Token Use

• Problem - LRMs have limited self correction capabilities - to know if chosen answer wrong early on, and to not doubt the actual right answer if it chooses

• Levels of LLM intelligence as I see it - Text generation based on pattern from data —> Informed response based on web search —> reasoning models based on logical algorithms —> Self correction and evaluation to come to conclusion —> Next phase

• Why dont LRMs benefit from explicit algorithms they have access to for high complex problems?

• Problem - obtaining high quality Chain of thought data very expensive because it so scarce

• The weird opposite correlation in thinking effort and complexity

  • Effort in reasoning/thinking should increase if problem is more complex, like we humans do —> but LRMs exhibit the opposite after a certain level
    • Maybe due to implemented roadblocks in efficiency that they give up and compromise?
    • Also observed that initially thinking tokens increase with complexity, but when it reachers a threshold when accuracy “collapses”, they reduce effort actively
    • Thing is, ample compute is available to actually use the tokens to use explicit algorithms to address the complex problems, but again, they “give up” early.

Evaluation Benchmarks, Token Inefficiency, and Personal Thoughts

• Evaluation benchmarks needs to change

  • Common ones are math and code problems
  • Apple introduces “Controllable Puzzle Environments” for manipulation of complexity

• The inefficiency problem

  • Simple problems - LRM get answer early, and use up unnecessary remaining tokens exploring other incorrect solutions (better-be-safe ahh)
  • Complex problems - LRM first explore incorrect solutions steadily, and then as they think arrive at a correct answer.
  • Higher complexity - Collapse in answer accuracy - “give up”

• Interesting thought from my side

• If LLMs are smart on easier problems, and spend remaining time and tokens exploring alternatives, but on complex problems go through their solutions, initially being incorrect, and then reach the correct one —> they are essentially using the same amount of tokens regardless (one scenario just to make sure a right answer, and one to get to the right answer).

  • Both scenarios involve understanding and explaining why incorrect solutions are…incorrect.
  • Its like on the SAT reading all the easier ones you get answer right away and you just “make sure” but on the harder ones you use process of elimination
  • This is probably why there is that inverse relationship between effort and problem complexity
    • Because once its beyond its threshold, and realizes more tokens will be needed than its usual, just gives up and does not even use explicit algorithms given to it to solve (Collapsing)
    • It looks like almost as if the LRM needs to use a set of tokens (certain value) and only that, not more not less
    • And when it recognises a problem beyond that, it uses less tokens than needed, and problem much within that complexity, uses more tokens than needed —→ converging to that specific value
    • Which means these LRM models are more token efficiency aware that they first determine if the problem falls close to how many tokens would be needed, and then does its analysis.
    • Like you would do on the SAT, where if a certain reading problem or the math problem at the very end is too hard (crazy amount of logical thinking needed), you just leave it, because it is going to take time to find a solution to that (the complex ones) and its not worth it.
      • You can approach the problem, from what you know (even if its seems impossible) by breaking it down and connecting the dots (pretty vague but think about it)
      • Honestly sometimes can be solved with first principles.
    • Obviously they made LRMs token aware to save costs and compute
      • But if you make the LRM less token aware its just gonna use up a lot of tokens in the thinking and screw up your token budget
      • What is the solution ?

Limitation: Exact Computation

• Also another big problem is they just cannot do exact computation - very much text based reasoning and pattern matching only (all you gotta do is just substitute in values brother)
  • Because apparently when researchers gave the solution algorithm for Tower of Hanoi puzzle, and when the LRM just needs to use it to find answer (performance improves), it apparently did not.