How does Tree of Thoughts improve on Chain of Thought?

What you need to know

The key idea is commitment. In CoT, each token is chosen once and never revisited. If step 2 picks a poor option, steps 3 to 10 build on it. A human solver would notice the dead end and try something else. CoT has no mechanism for that.

The three improvements

• Exploration means good options are not lost just because the first sample missed them.
• Evaluation puts effort on promising branches and prunes hopeless ones early.
• Backtracking turns a dead end into a cheap detour instead of a wrong final answer.

The evidence

In the Tree of Thoughts paper (Yao et al., 2023), GPT-4 with chain of thought solved 4% of Game of 24 puzzles, while ToT with a breadth of 5 solved 74%. Gains were also reported for creative writing with constraints and mini crosswords. These are problems where early choices strongly limit later ones.

Why the gap is smaller in 2026

Reasoning models are trained to explore, notice mistakes and revise inside their hidden thinking. That captures much of ToT's benefit in a single call. What a single call still lacks is an external, exact evaluator. So ToT-style search still wins when you can check partial or complete answers in code — tests for code, a solver for schedules, a validator for constraints.

A practical middle ground

Best-of-N with a verifier: generate N complete candidates in parallel, check each with code, keep the best. It is simpler than a full tree and captures most of the benefit when complete answers are cheap to check.

A real-life example

An e-commerce team generates SEO titles for product pages with hard rules: at most 60 characters, brand first, include the size, no words from a banned list, and unique across the catalogue.

• CoT, one call: the model reasons through the rules and writes one title. About 18% break a rule — usually length or uniqueness — and the whole title is regenerated.
• Tree-style search: the model proposes 4 openings after the brand; code checks length so far and the banned list; the best 2 are extended with 3 endings each; code checks uniqueness against the catalogue index and picks the shortest valid one.

Branch A: "HomeKart 5 L Steel Pressure Cooker, Induction" (45 chars) validBranch B: "HomeKart Best Pressure Cooker..." banned word "Best" -> pruned

Rule failures fall to under 1%, at about 8 calls per title. For 10,000 titles run overnight in a batch job, the team accepts the cost.

Follow-up questions to expect

• "When is ToT not worth it?" — When a single reasoning call already succeeds most of the time, when latency matters, or when there is no reliable way to evaluate partial answers.
• "What evaluator would you use?" — Code wherever possible (tests, rules, solvers); a model-based rating only when nothing exact exists, validated on labelled cases.
• "How does best-of-N relate?" — It is a flat, one-level tree: many complete candidates, one verification step. Often the best cost-to-benefit option.